+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210905/B1f --output-run DEEP/20210905/B1f/science#step0/20240316T110805Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.2563 seconds; current memory usage: 0.179 Gibyte, delta: 0.043 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0035 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0067 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/pipeline_orig.qgraph DEEP/20210905/B1f/science#step0/20240316T110805Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.6747 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.2296 seconds; current memory usage: 0.272 Gibyte, delta: 0.093 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.268 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0134 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.268 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 372 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210905_B1f_science#step0_20240316T110805Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0810 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.268 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 374 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1998 seconds; current memory usage: 0.274 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.274 Gibyte (main), 0.268 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: b1089c9e-7e95-41e9-b1aa-2792bf798b2c parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-16 11:08:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-16 11:08:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 11:08:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-16 11:08:29 proc_lsst.multi:170 [INFO] [multi] job_id 15207 proc_lsst.multi INFO: [multi] job_id 15207 2024-03-16 11:08:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:29 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 15207 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 15207 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 15207 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash ed92698accb4849085ee0a805e933095 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1f49f0ff-cec1-487d-926d-2f9cea569413_isrForCrosstalkSources_1029418_48.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1f49f0ff-cec1-487d-926d-2f9cea569413_isrForCrosstalkSources_1029418_48.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 648ed0970cc65a5a9167e0943a927bce parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bbfe794c-baea-46bd-b82f-2cc1ac3ca9ee_isrForCrosstalkSources_1029418_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bbfe794c-baea-46bd-b82f-2cc1ac3ca9ee_isrForCrosstalkSources_1029418_62.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash fc32b568dcde15a82128be69e064dff0 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/4d04e5d0-6b86-4ada-b596-fcb517347123_isrForCrosstalkSources_1029419_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/4d04e5d0-6b86-4ada-b596-fcb517347123_isrForCrosstalkSources_1029419_43.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash ff76666de52b286c1d708d98fcd33d0b parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/796c6a06-9931-4c1e-b803-b13366bb16fb_isrForCrosstalkSources_1029418_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/796c6a06-9931-4c1e-b803-b13366bb16fb_isrForCrosstalkSources_1029418_16.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash b46d90550892f6e87144d5b7230b7ef9 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/94e1dd67-ea60-4b6e-88bc-16a82879f481_isrForCrosstalkSources_1029418_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/94e1dd67-ea60-4b6e-88bc-16a82879f481_isrForCrosstalkSources_1029418_13.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 7f877a99a07c611c21b6070cc651b4b9 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93758a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/bdf897ad-ec5c-43e9-b461-41b821d0e0b1_isrForCrosstalkSources_1029419_57.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/bdf897ad-ec5c-43e9-b461-41b821d0e0b1_isrForCrosstalkSources_1029419_57.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 216d3483f124428b327c9027569b5115 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7e19de78-d857-404d-8524-db150f96bda7_isrForCrosstalkSources_1029419_29.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7e19de78-d857-404d-8524-db150f96bda7_isrForCrosstalkSources_1029419_29.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 3e213f62b473a9cf3f707d27e7ecc1fa parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9374c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03886027-e768-4e1f-854c-fd102e81c73c_isrForCrosstalkSources_1029420_26.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03886027-e768-4e1f-854c-fd102e81c73c_isrForCrosstalkSources_1029420_26.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash d29a67e1e53543b6367ba54b8984b2e5 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ad347104-d1cf-4cf1-ae1f-8f0aad801889_isrForCrosstalkSources_1029424_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ad347104-d1cf-4cf1-ae1f-8f0aad801889_isrForCrosstalkSources_1029424_21.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 4152f319c46e7cab97b6ea9c00f4868f parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93754e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/eadf54b9-1b20-4e72-9c6e-7052cf305c76_isrForCrosstalkSources_1029424_42.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/eadf54b9-1b20-4e72-9c6e-7052cf305c76_isrForCrosstalkSources_1029424_42.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 9bea79565f7dcf03ae8005dfadb90f89 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93759e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/db807573-57af-4e38-85fb-fa3ce0e2e64a_isrForCrosstalkSources_1029419_20.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/db807573-57af-4e38-85fb-fa3ce0e2e64a_isrForCrosstalkSources_1029419_20.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 3e3e787c339b52bd4da71888cc7e47e4 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8fbdfedb-a9d8-4840-b05d-cb1fb8b91e22_isrForCrosstalkSources_1029419_5.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8fbdfedb-a9d8-4840-b05d-cb1fb8b91e22_isrForCrosstalkSources_1029419_5.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 91ce59facdf5a4adb624a09054362186 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/56cfcab2-5675-421e-8cb5-d51a397ac745_isrForCrosstalkSources_1029426_54.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/56cfcab2-5675-421e-8cb5-d51a397ac745_isrForCrosstalkSources_1029426_54.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash d061fe522ed1ce5c92d3b0035832977e parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a0da7dde-da8c-4980-b9c3-8e7567fb5c22_isrForCrosstalkSources_1029424_2.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a0da7dde-da8c-4980-b9c3-8e7567fb5c22_isrForCrosstalkSources_1029424_2.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash 670fdd808dba65bae9c922494c15af8f parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4a6398af-cbe6-40d6-bd62-eb16e25d8189_isrForCrosstalkSources_1029425_30.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4a6398af-cbe6-40d6-bd62-eb16e25d8189_isrForCrosstalkSources_1029425_30.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash e5c40dc2097f2b2b6523a1d64fb56782 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9375ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f7a02712-f5e2-4b3e-8a4b-a150a99eee1f_isrForCrosstalkSources_1029420_34.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f7a02712-f5e2-4b3e-8a4b-a150a99eee1f_isrForCrosstalkSources_1029420_34.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 25f8dc0cd9f76830e025a415436ecdcc parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93763e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b5feb600-7c6e-4a04-b7bf-c3dfd1a1a802_isrForCrosstalkSources_1029418_36.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b5feb600-7c6e-4a04-b7bf-c3dfd1a1a802_isrForCrosstalkSources_1029418_36.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 12eab8f74e970734440cd89b76c39550 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93765c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/811225b2-f7a1-4993-8b4a-68a0d059eba9_isrForCrosstalkSources_1029420_55.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/811225b2-f7a1-4993-8b4a-68a0d059eba9_isrForCrosstalkSources_1029420_55.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 9d5b528555d283e711da7ecae9c62961 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd2f294-993c-4daa-b2d7-37322c4b2ad5_isrForCrosstalkSources_1029420_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd2f294-993c-4daa-b2d7-37322c4b2ad5_isrForCrosstalkSources_1029420_13.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 5a26a128a09f6b443b5b81f06a860bf0 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87413a7b-a034-4b1b-945d-ce0f07616f7a_isrForCrosstalkSources_1029424_31.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87413a7b-a034-4b1b-945d-ce0f07616f7a_isrForCrosstalkSources_1029424_31.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash d4e96d11b7185605a6499e8466a06585 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93762a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/cf44b1b2-3c8d-470f-bfbe-0c41ed783fc9_isrForCrosstalkSources_1029426_21.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/cf44b1b2-3c8d-470f-bfbe-0c41ed783fc9_isrForCrosstalkSources_1029426_21.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 818a3e91b7f4f420a887195bff570d70 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/7f2d354a-ba99-4537-a8ca-6b28a43466a2_isrForCrosstalkSources_1029420_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/7f2d354a-ba99-4537-a8ca-6b28a43466a2_isrForCrosstalkSources_1029420_17.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 2ef065e34d60dcc3247679cc7ac9b27a parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/633b1afe-d5a6-4518-b445-8de384b96fa7_isrForCrosstalkSources_1029426_18.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/633b1afe-d5a6-4518-b445-8de384b96fa7_isrForCrosstalkSources_1029426_18.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash f4ae10c24e119adb52cf0cd5dcca586c parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93767a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/4e470592-8e71-45db-8b2d-d431acd1d9a3_isrForCrosstalkSources_1029424_45.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/4e470592-8e71-45db-8b2d-d431acd1d9a3_isrForCrosstalkSources_1029424_45.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash a1c3276ddb745ff3311bf3444e9f6b2f parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93b1afab-1516-4106-8846-1c0e8dfdd83a_isrForCrosstalkSources_1029419_35.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93b1afab-1516-4106-8846-1c0e8dfdd83a_isrForCrosstalkSources_1029419_35.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash cfc2f580254e40eaaf1ac1103ec89016 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/20a9f84a-d26b-4fb2-8034-dba1c57e6300_isrForCrosstalkSources_1029418_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/20a9f84a-d26b-4fb2-8034-dba1c57e6300_isrForCrosstalkSources_1029418_61.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash fbbfbc4abeca422bd077d9639adf2417 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93768e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0a0d00b4-a61f-4940-8f63-34ba7e7d0697_isrForCrosstalkSources_1029424_36.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0a0d00b4-a61f-4940-8f63-34ba7e7d0697_isrForCrosstalkSources_1029424_36.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 7a4e84f73e7279139180a094befa695d parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7d11ebcd-b010-48da-8285-fb5db08f19ae_isrForCrosstalkSources_1029426_30.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7d11ebcd-b010-48da-8285-fb5db08f19ae_isrForCrosstalkSources_1029426_30.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 769e99723415a9530a68dc20aa67b779 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/631c919a-a540-4574-bbbe-7e332dc484e4_isrForCrosstalkSources_1029424_7.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/631c919a-a540-4574-bbbe-7e332dc484e4_isrForCrosstalkSources_1029424_7.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash d7402aeb76ee949edc742101e5d0efb6 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/847b842f-23ac-4a75-b8a9-cd5257781cd6_isrForCrosstalkSources_1029418_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/847b842f-23ac-4a75-b8a9-cd5257781cd6_isrForCrosstalkSources_1029418_2.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 10e061c8eb271d704072c0e0255b4f24 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/3f20170a-96f2-49e2-8487-4763f16f4982_isrForCrosstalkSources_1029419_3.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/3f20170a-96f2-49e2-8487-4763f16f4982_isrForCrosstalkSources_1029419_3.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash e999e871774ae3f3635db0055e5b9461 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3931456c-f10b-439a-b5b7-ea3659f613c3_isrForCrosstalkSources_1029424_58.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3931456c-f10b-439a-b5b7-ea3659f613c3_isrForCrosstalkSources_1029424_58.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 39df00fd15ea7cb41d2fea30c903f51b parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a5802c0a-f0d7-48cc-b741-68985452ba18_isrForCrosstalkSources_1029424_35.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a5802c0a-f0d7-48cc-b741-68985452ba18_isrForCrosstalkSources_1029424_35.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 811153df4f90b24f984e67ef23b61502 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2645ecfb-8c6b-4678-a1bf-d7f43334dcd7_isrForCrosstalkSources_1029418_46.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2645ecfb-8c6b-4678-a1bf-d7f43334dcd7_isrForCrosstalkSources_1029418_46.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 70bbd1bb91e8657a9ae5fa3233ac3312 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f2999621-497e-4688-af59-291c2ed5bc2a_isrForCrosstalkSources_1029418_22.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f2999621-497e-4688-af59-291c2ed5bc2a_isrForCrosstalkSources_1029418_22.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 29391f2ddd44ce8d1841577c871c76ca parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9104638f-5b4d-40a3-8206-b06875361b6f_isrForCrosstalkSources_1029426_49.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9104638f-5b4d-40a3-8206-b06875361b6f_isrForCrosstalkSources_1029426_49.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash db57fb42b41c4d9143451c90a5b8dff3 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4421e02b-ed01-47e2-90f8-ae606251d19a_isrForCrosstalkSources_1029426_60.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4421e02b-ed01-47e2-90f8-ae606251d19a_isrForCrosstalkSources_1029426_60.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 4305dfa3c839a97de8917c279ac16249 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93771a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/7e184b7e-5750-45cb-88ae-2aa0cbff8248_isrForCrosstalkSources_1029418_18.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/7e184b7e-5750-45cb-88ae-2aa0cbff8248_isrForCrosstalkSources_1029418_18.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 2b646b7b3133c79e7d77ac53054e906f parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d97c063c-0cac-4cb4-915f-2422036b5ae9_isrForCrosstalkSources_1029419_38.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d97c063c-0cac-4cb4-915f-2422036b5ae9_isrForCrosstalkSources_1029419_38.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 35499d557048a9336b242ebe4e6609d5 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/216c3002-c5bd-493d-addf-15f20850f6ea_isrForCrosstalkSources_1029419_46.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/216c3002-c5bd-493d-addf-15f20850f6ea_isrForCrosstalkSources_1029419_46.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash a635b1987e831c7a1464dca7f7d7c432 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c4382375-d083-4d70-90c4-ef7576bffd2d_isrForCrosstalkSources_1029426_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c4382375-d083-4d70-90c4-ef7576bffd2d_isrForCrosstalkSources_1029426_47.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 0cca929f640725bb8036597c7a63dcf4 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c6f4633c-605a-4c6a-9677-2b08d0177d70_isrForCrosstalkSources_1029418_50.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c6f4633c-605a-4c6a-9677-2b08d0177d70_isrForCrosstalkSources_1029418_50.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 602f038508a501820090738d7756a4e9 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e8fddf3b-5e35-4a72-b99a-b791d9824d27_isrForCrosstalkSources_1029426_11.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e8fddf3b-5e35-4a72-b99a-b791d9824d27_isrForCrosstalkSources_1029426_11.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 9a518db878a9ad022b353531cfe5c239 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f51c09fa-ce0d-423b-80ce-1c594c52cedf_isrForCrosstalkSources_1029426_20.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f51c09fa-ce0d-423b-80ce-1c594c52cedf_isrForCrosstalkSources_1029426_20.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash db681d53f5e3a88891f6394b92b57eda parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/05b620d7-f6bd-4c1f-9375-cc3cf4509802_isrForCrosstalkSources_1029418_23.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/05b620d7-f6bd-4c1f-9375-cc3cf4509802_isrForCrosstalkSources_1029418_23.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash d8182c4d8d5b651b934b64b612185565 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/02ba2d26-ffbd-44ad-8aee-b8f73bda1398_isrForCrosstalkSources_1029426_4.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/02ba2d26-ffbd-44ad-8aee-b8f73bda1398_isrForCrosstalkSources_1029426_4.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 94af40601435f62be768d31a38707eb7 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b187ed86-4268-4358-b96d-1a5f3466e61b_isrForCrosstalkSources_1029424_13.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b187ed86-4268-4358-b96d-1a5f3466e61b_isrForCrosstalkSources_1029424_13.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash f4425525d1fd3f41e9cda61af9ea1d48 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93779c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/72ab27b8-1407-458f-8627-83c8b3c0ec75_isrForCrosstalkSources_1029420_43.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/72ab27b8-1407-458f-8627-83c8b3c0ec75_isrForCrosstalkSources_1029420_43.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 918211de4671497dc2a351b9ef34b393 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/5799789a-3975-4d62-8830-f67f1bdedde8_isrForCrosstalkSources_1029425_27.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/5799789a-3975-4d62-8830-f67f1bdedde8_isrForCrosstalkSources_1029425_27.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash f33c6705493a59bbeda4caef2ab08560 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93777e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd130870-16e8-4622-b63e-70579b3837bf_isrForCrosstalkSources_1029425_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd130870-16e8-4622-b63e-70579b3837bf_isrForCrosstalkSources_1029425_26.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash b265e3fc8bc178fedacdd8ae0b9f54f7 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7592382b-2dbd-4770-8382-363c96477d74_isrForCrosstalkSources_1029425_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7592382b-2dbd-4770-8382-363c96477d74_isrForCrosstalkSources_1029425_34.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash fb7e1cab447bbfe9bc79723bf3a23772 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c680fe76-ebdd-449d-9fa0-4c20c857fbfc_isrForCrosstalkSources_1029424_18.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c680fe76-ebdd-449d-9fa0-4c20c857fbfc_isrForCrosstalkSources_1029424_18.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 0a05f193a2abdb4cd9b911ed11815e8a parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93776a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/808af997-6e10-4a5c-a704-a4a3235ad4f4_isrForCrosstalkSources_1029426_17.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/808af997-6e10-4a5c-a704-a4a3235ad4f4_isrForCrosstalkSources_1029426_17.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 9e59582a4bd59a4895fddddb9a516810 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8f993352-e07d-40ea-9097-8a90d32819a7_isrForCrosstalkSources_1029420_53.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8f993352-e07d-40ea-9097-8a90d32819a7_isrForCrosstalkSources_1029420_53.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 61acb379031e89d72526360b5006b0a4 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93774c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/096d1c6b-fb9a-4e21-bbf6-6116d4c21814_isrForCrosstalkSources_1029419_51.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/096d1c6b-fb9a-4e21-bbf6-6116d4c21814_isrForCrosstalkSources_1029419_51.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 06684a30bd419eb007e40cf00f8c66bb parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/402b3493-39a0-4b3f-a9d6-12dbe638b725_isrForCrosstalkSources_1029425_12.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/402b3493-39a0-4b3f-a9d6-12dbe638b725_isrForCrosstalkSources_1029425_12.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 444d6175b3b8f17d43d310dcb470ce66 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c042330c-32b2-4c5e-94a4-21ee97eb161d_isrForCrosstalkSources_1029424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c042330c-32b2-4c5e-94a4-21ee97eb161d_isrForCrosstalkSources_1029424_55.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 0b16ead9c54aaa21856bbca4e518c94f parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0c5d61e8-1a91-482f-90bf-77299c1aafc9_isrForCrosstalkSources_1029424_32.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0c5d61e8-1a91-482f-90bf-77299c1aafc9_isrForCrosstalkSources_1029424_32.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 6b57da4797431a98b241aa945d795c70 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93772e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/6c532ab1-7b9a-4bf4-8843-158a9115c989_isrForCrosstalkSources_1029418_54.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/6c532ab1-7b9a-4bf4-8843-158a9115c989_isrForCrosstalkSources_1029418_54.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 99aa2b6def1427f552da6bb72d05db20 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a1e5a3c5-2704-490b-96af-f44ebbc9b8c5_isrForCrosstalkSources_1029418_1.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a1e5a3c5-2704-490b-96af-f44ebbc9b8c5_isrForCrosstalkSources_1029418_1.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 31e6b62455d619e1db3fcf1f4aa85d15 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9376520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3f259590-43f2-4904-b482-f66f7b57a11d_isrForCrosstalkSources_1029424_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3f259590-43f2-4904-b482-f66f7b57a11d_isrForCrosstalkSources_1029424_60.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash bb2d57d23a662b09f5d523cc270e9b52 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9cd66295-10e0-4644-aa6a-718e88f7bf08_isrForCrosstalkSources_1029418_5.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9cd66295-10e0-4644-aa6a-718e88f7bf08_isrForCrosstalkSources_1029418_5.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 3e6465a90652391b1d4849143f25e1c0 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93c54dff-bc02-4a01-8fc7-7d083a175925_isrForCrosstalkSources_1029419_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93c54dff-bc02-4a01-8fc7-7d083a175925_isrForCrosstalkSources_1029419_26.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash a4c363b3ed17fe4f9bc0fbbb17a2fc80 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/1eccc6b6-31ed-4ffc-8187-b1e816a5d170_isrForCrosstalkSources_1029420_42.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/1eccc6b6-31ed-4ffc-8187-b1e816a5d170_isrForCrosstalkSources_1029420_42.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 011d8891ed92d487634ed4835406384a parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7b27a528-9110-46b2-a0cb-7e774b86f3b5_isrForCrosstalkSources_1029426_42.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7b27a528-9110-46b2-a0cb-7e774b86f3b5_isrForCrosstalkSources_1029426_42.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash eadd139dfded25d0c6823e43f10511f8 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/76f7ed00-32ea-4058-84c7-6f2cdc5a3e88_isrForCrosstalkSources_1029418_57.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/76f7ed00-32ea-4058-84c7-6f2cdc5a3e88_isrForCrosstalkSources_1029418_57.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash ea63710649086af04f658d0d5bf9ae79 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/622a937d-fbec-481e-9f81-92bc9cbb405c_isrForCrosstalkSources_1029425_37.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/622a937d-fbec-481e-9f81-92bc9cbb405c_isrForCrosstalkSources_1029425_37.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 9c4534b5edda2964d34b357c9d81d9fb parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/410a300b-ff87-4029-92bf-0d9bfb64600f_isrForCrosstalkSources_1029419_32.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/410a300b-ff87-4029-92bf-0d9bfb64600f_isrForCrosstalkSources_1029419_32.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 43f84b991c665e56ba9f768157c75e2b parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c9c2d679-3780-4039-a462-b356c5092f77_isrForCrosstalkSources_1029420_56.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c9c2d679-3780-4039-a462-b356c5092f77_isrForCrosstalkSources_1029420_56.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash dcc46cec871e8e355ff2430330abfcd5 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3bf0eaab-b31b-49e4-97f6-ae18c4277246_isrForCrosstalkSources_1029418_15.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3bf0eaab-b31b-49e4-97f6-ae18c4277246_isrForCrosstalkSources_1029418_15.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash b5e81b05653eb5ec3dfa48122f527804 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7759b3fd-3ff6-4d17-8280-0db966bfb0aa_isrForCrosstalkSources_1029425_43.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7759b3fd-3ff6-4d17-8280-0db966bfb0aa_isrForCrosstalkSources_1029425_43.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 3095ad49eedfd7ce84e7f824ebaffd8e parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/240de32f-129b-41fb-8090-d584a511e1a0_isrForCrosstalkSources_1029419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/240de32f-129b-41fb-8090-d584a511e1a0_isrForCrosstalkSources_1029419_13.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 2caeadd4200762a255d6c2bd7dce2393 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/143e882f-792f-481f-9950-4a85fb870888_isrForCrosstalkSources_1029424_15.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/143e882f-792f-481f-9950-4a85fb870888_isrForCrosstalkSources_1029424_15.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 4cae50125a331a14e85e136cb557b70a parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/74cb822c-a544-4912-9416-a617378ab0d9_isrForCrosstalkSources_1029418_11.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/74cb822c-a544-4912-9416-a617378ab0d9_isrForCrosstalkSources_1029418_11.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash f7a625ab7161d3ebdf1f15e528b05fc8 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2709f67c-66b3-4312-adf0-bde1057ce0ea_isrForCrosstalkSources_1029425_31.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2709f67c-66b3-4312-adf0-bde1057ce0ea_isrForCrosstalkSources_1029425_31.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash f5b9a0d693ec3da4f8ab5f078022e94b parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cc473ed9-e7ff-4745-8acb-38b1d503f736_isrForCrosstalkSources_1029424_1.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cc473ed9-e7ff-4745-8acb-38b1d503f736_isrForCrosstalkSources_1029424_1.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash f26fa338fbab78e9fe055d7f8089c5ec parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6eb2923a-425d-4a36-b4e2-8121e105efc5_isrForCrosstalkSources_1029425_9.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6eb2923a-425d-4a36-b4e2-8121e105efc5_isrForCrosstalkSources_1029425_9.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 2eb6b4864dd2a91f28b05fdf044e8c7a parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/942a37c5-6d27-480e-80eb-5dadcb203904_isrForCrosstalkSources_1029424_57.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/942a37c5-6d27-480e-80eb-5dadcb203904_isrForCrosstalkSources_1029424_57.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 724b3dbe01ef8a9ca391180a701bf0c4 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d47d3f6a-6ae3-4463-b35e-cabe60fdb651_isrForCrosstalkSources_1029424_51.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d47d3f6a-6ae3-4463-b35e-cabe60fdb651_isrForCrosstalkSources_1029424_51.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 4f9cd725bd2cbdf78a6add41e118a1e3 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/2ae9c992-1f6b-4397-9244-d16541a3c4e9_isrForCrosstalkSources_1029424_24.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/2ae9c992-1f6b-4397-9244-d16541a3c4e9_isrForCrosstalkSources_1029424_24.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash cc519a18a74268085468f31d5780e98e parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f0542578-842d-432a-ab0e-45ec0f860020_isrForCrosstalkSources_1029420_6.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f0542578-842d-432a-ab0e-45ec0f860020_isrForCrosstalkSources_1029420_6.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash a938c3a854fe53f982936b0c8a273c63 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03fbbdaa-4201-4f18-9cfd-b8cfdf9248ed_isrForCrosstalkSources_1029420_35.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03fbbdaa-4201-4f18-9cfd-b8cfdf9248ed_isrForCrosstalkSources_1029420_35.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 592c68720b31e68338c9f42b31a010c2 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c48d3481-b1ea-4ffe-b972-b4fef90b0991_isrForCrosstalkSources_1029419_40.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c48d3481-b1ea-4ffe-b972-b4fef90b0991_isrForCrosstalkSources_1029419_40.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 9c0d35849743847fecd1194bd2afc72c parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7f00980f-1980-4c1c-b787-96ce599240c5_isrForCrosstalkSources_1029419_18.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7f00980f-1980-4c1c-b787-96ce599240c5_isrForCrosstalkSources_1029419_18.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 5d0c0b0c6289a89e8f9d8ad6879c822a parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/f966c98b-a75d-4ed3-ac0d-675b2ea3d2f3_isrForCrosstalkSources_1029419_53.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/f966c98b-a75d-4ed3-ac0d-675b2ea3d2f3_isrForCrosstalkSources_1029419_53.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 47a37c8cbf5076ebbabebdedfdf0a10e parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1c38d82c-cf82-4368-8299-3ec1d1f34e93_isrForCrosstalkSources_1029424_12.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1c38d82c-cf82-4368-8299-3ec1d1f34e93_isrForCrosstalkSources_1029424_12.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 18ae69c9a9683414151912fa1d1e88bc parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1b02517c-94be-46f3-b1da-e0a1dcb1e255_isrForCrosstalkSources_1029424_41.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1b02517c-94be-46f3-b1da-e0a1dcb1e255_isrForCrosstalkSources_1029424_41.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 3ee1a90672e415725c7b43c7f7a51fbd parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/da52466c-fca0-49b9-afc8-b28a823c93f6_isrForCrosstalkSources_1029424_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/da52466c-fca0-49b9-afc8-b28a823c93f6_isrForCrosstalkSources_1029424_20.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash d06f2996dd27fb4fcf7050b86a8cc09b parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/64441d16-e240-4a0f-9f0a-73184757d16e_isrForCrosstalkSources_1029424_9.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/64441d16-e240-4a0f-9f0a-73184757d16e_isrForCrosstalkSources_1029424_9.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 58a2564ef784f3d01489b1786bd21cc4 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/61860506-8c30-4163-8545-df9c6cfabed9_isrForCrosstalkSources_1029426_46.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/61860506-8c30-4163-8545-df9c6cfabed9_isrForCrosstalkSources_1029426_46.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 13d5d54e79ced6ee653f3857d5911930 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3759c4b9-e7f1-496e-a3b0-e37451cd4692_isrForCrosstalkSources_1029418_26.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3759c4b9-e7f1-496e-a3b0-e37451cd4692_isrForCrosstalkSources_1029418_26.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 7b7c47f84bfec2b7b9605d7b4681c4f2 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7c9917c8-80b1-46d2-b39f-f88a90fddb63_isrForCrosstalkSources_1029425_1.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7c9917c8-80b1-46d2-b39f-f88a90fddb63_isrForCrosstalkSources_1029425_1.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 9ba5b418e0fc824f402578406326531d parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6e6f29f8-20fb-4bdd-bd2a-e5dc21f82b12_isrForCrosstalkSources_1029425_50.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6e6f29f8-20fb-4bdd-bd2a-e5dc21f82b12_isrForCrosstalkSources_1029425_50.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 701d904d97328be0eff13d40eb893c3b parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c026084c-20dc-45f6-a8f6-22ccd9f0ca38_isrForCrosstalkSources_1029424_38.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c026084c-20dc-45f6-a8f6-22ccd9f0ca38_isrForCrosstalkSources_1029424_38.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash d08e6e7d0cfe5b7c499bf8e220875667 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/d16c4f16-6f8e-4c0d-a962-81b069ef4983_isrForCrosstalkSources_1029420_40.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/d16c4f16-6f8e-4c0d-a962-81b069ef4983_isrForCrosstalkSources_1029420_40.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash fc82765efa5fa984172174387aef1f96 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a9fcf7bc-24bd-437e-ad53-a6a337b12e13_isrForCrosstalkSources_1029425_36.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a9fcf7bc-24bd-437e-ad53-a6a337b12e13_isrForCrosstalkSources_1029425_36.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 1ed33836e8a9938f0836d9ad27afacfc parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/59924cdb-6824-4977-ad4e-95425fe62e60_isrForCrosstalkSources_1029426_15.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/59924cdb-6824-4977-ad4e-95425fe62e60_isrForCrosstalkSources_1029426_15.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 196e83ede0031f93fe087a58c6f649d7 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/40b1f608-b2fb-4459-9d82-027a8ace2756_isrForCrosstalkSources_1029419_14.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/40b1f608-b2fb-4459-9d82-027a8ace2756_isrForCrosstalkSources_1029419_14.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash c65dcc886e4b9e2f8b3f6ccacbcc390b parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8acee2d4-a270-4b7c-8523-9ee65ff9a635_isrForCrosstalkSources_1029424_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8acee2d4-a270-4b7c-8523-9ee65ff9a635_isrForCrosstalkSources_1029424_48.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 65f418b9234fa34a7d9596407961b091 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/a6297901-7f69-42ba-b03e-10093e585c43_isrForCrosstalkSources_1029426_35.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/a6297901-7f69-42ba-b03e-10093e585c43_isrForCrosstalkSources_1029426_35.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 390bbd6383571eba1c0dcf829619d0e9 parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c40042c0-a78b-4bd7-a42f-8aaa42f2c35d_isrForCrosstalkSources_1029424_14.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c40042c0-a78b-4bd7-a42f-8aaa42f2c35d_isrForCrosstalkSources_1029424_14.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash f3f34589643f6288c25f78ec50223633 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/eb53e863-1cd1-4072-ace7-e20d642a5e7c_isrForCrosstalkSources_1029426_52.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/eb53e863-1cd1-4072-ace7-e20d642a5e7c_isrForCrosstalkSources_1029426_52.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 47944b4e7feae9c012e4852be2445f71 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/852ce436-aec8-48b6-9fdb-d92e122ce483_isrForCrosstalkSources_1029419_50.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/852ce436-aec8-48b6-9fdb-d92e122ce483_isrForCrosstalkSources_1029419_50.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash b9df2653ef892eb43bbb161adde2c721 parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bf8617a5-0575-46af-8f8c-7f1abe0bc4b6_isrForCrosstalkSources_1029424_37.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bf8617a5-0575-46af-8f8c-7f1abe0bc4b6_isrForCrosstalkSources_1029424_37.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 5ce99faa145cbd9975b5f6afee4c6bd4 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9c5ac8d1-d0c0-4ad8-816f-73afc481fb81_isrForCrosstalkSources_1029418_37.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9c5ac8d1-d0c0-4ad8-816f-73afc481fb81_isrForCrosstalkSources_1029418_37.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 5a7bc7797dd7c4a94a00d42825d875a8 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ebf4b53e-bd7d-4af2-bc6f-98148f426a9e_isrForCrosstalkSources_1029425_28.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ebf4b53e-bd7d-4af2-bc6f-98148f426a9e_isrForCrosstalkSources_1029425_28.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash b61e208ab1398a5c7d9cb60d12cf937a parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/57624f3e-9976-41db-9136-14dfc05a1b83_isrForCrosstalkSources_1029419_6.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/57624f3e-9976-41db-9136-14dfc05a1b83_isrForCrosstalkSources_1029419_6.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash c909789fde939502f4b1feb82edd3467 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7dc23b0e-4d11-456e-a62b-725b4f0d76e7_isrForCrosstalkSources_1029426_61.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7dc23b0e-4d11-456e-a62b-725b4f0d76e7_isrForCrosstalkSources_1029426_61.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 697cd56696ab8717fdf96f2d48a6619e parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0844495a-a046-4c0a-8e16-31ec18d51a29_isrForCrosstalkSources_1029419_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0844495a-a046-4c0a-8e16-31ec18d51a29_isrForCrosstalkSources_1029419_47.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 67155995f0767d9bc7c7d6e359a5f7a1 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/53ea78e8-42ba-4e56-b5c2-a20365cbf7c4_isrForCrosstalkSources_1029418_43.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/53ea78e8-42ba-4e56-b5c2-a20365cbf7c4_isrForCrosstalkSources_1029418_43.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 5a5a6463c111be0acd187096d31449fc parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/71e10703-60a4-4fca-bfda-b011846dcc1f_isrForCrosstalkSources_1029420_23.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/71e10703-60a4-4fca-bfda-b011846dcc1f_isrForCrosstalkSources_1029420_23.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 43c3e4395cde9cbef370516c92736e5c parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c497c9b0-4e95-4eb1-8e9a-4df9f94604d5_isrForCrosstalkSources_1029419_30.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c497c9b0-4e95-4eb1-8e9a-4df9f94604d5_isrForCrosstalkSources_1029419_30.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 850e0f5eee52f7dcbbdd049b25f703fc parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e3b88e82-8c59-479d-b16e-8db5e742dd6c_isrForCrosstalkSources_1029425_6.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e3b88e82-8c59-479d-b16e-8db5e742dd6c_isrForCrosstalkSources_1029425_6.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash d68af3f22c04486a46c05845633b090e parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/b7c15585-24f8-4e14-b051-2c401f13a1bf_isrForCrosstalkSources_1029426_13.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/b7c15585-24f8-4e14-b051-2c401f13a1bf_isrForCrosstalkSources_1029426_13.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 5842960f03c3fc6123515f449e64e901 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/973b6ec9-f725-4add-97db-d32ff2282b7f_isrForCrosstalkSources_1029426_62.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/973b6ec9-f725-4add-97db-d32ff2282b7f_isrForCrosstalkSources_1029426_62.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 684214d7b8147e22055709a0b7e4fc29 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8b69c480-874d-4a14-a6a0-c4c796dee1db_isrForCrosstalkSources_1029420_57.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8b69c480-874d-4a14-a6a0-c4c796dee1db_isrForCrosstalkSources_1029420_57.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 63d06db81e7312f23979f40a945c0148 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/da03c397-507d-4db3-a4eb-5f230bbb3cd1_isrForCrosstalkSources_1029418_10.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/da03c397-507d-4db3-a4eb-5f230bbb3cd1_isrForCrosstalkSources_1029418_10.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 3c281b0fd164298d83082b5658853f9f parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/715c7c68-7228-4c1b-b1cd-940f7fabe9ac_isrForCrosstalkSources_1029420_37.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/715c7c68-7228-4c1b-b1cd-940f7fabe9ac_isrForCrosstalkSources_1029420_37.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 75af5e978a4784a4c85d05da5be86ed7 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/4408e0e5-400f-4a56-99a4-ca4b421597e4_isrForCrosstalkSources_1029418_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/4408e0e5-400f-4a56-99a4-ca4b421597e4_isrForCrosstalkSources_1029418_29.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash dcf3700df04cadd336024604904069ff parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9fa60787-8e60-4e34-9641-53bd9ffbb0f0_isrForCrosstalkSources_1029425_22.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9fa60787-8e60-4e34-9641-53bd9ffbb0f0_isrForCrosstalkSources_1029425_22.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 7bff606b0f05de8279dafa03194dd74a parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/71ae7561-9de1-4908-ba46-e9701c994a5d_isrForCrosstalkSources_1029418_17.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/71ae7561-9de1-4908-ba46-e9701c994a5d_isrForCrosstalkSources_1029418_17.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash e6a0209ccfe530774173f0ae1f2bd75c parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ae94d220-b028-4235-8b49-ab4c5a8bb8ec_isrForCrosstalkSources_1029424_19.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ae94d220-b028-4235-8b49-ab4c5a8bb8ec_isrForCrosstalkSources_1029424_19.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 3d86cea8625ae9a71785a3ac7ed7e854 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e4d2a9ad-3dfa-405f-a537-f984f3f7278f_isrForCrosstalkSources_1029424_59.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e4d2a9ad-3dfa-405f-a537-f984f3f7278f_isrForCrosstalkSources_1029424_59.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 66e7881dba7571086e750d730fe0c573 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f69a3fa2-b33f-430d-bcae-1f4075255a21_isrForCrosstalkSources_1029425_15.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f69a3fa2-b33f-430d-bcae-1f4075255a21_isrForCrosstalkSources_1029425_15.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 4d453a8c39b93b783499d95258b98af9 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/58dda0bc-4f56-4cd0-921b-46202ce54c3f_isrForCrosstalkSources_1029425_39.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/58dda0bc-4f56-4cd0-921b-46202ce54c3f_isrForCrosstalkSources_1029425_39.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash cc2bcf32f248c0116729dfefee5056d2 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0f014ff3-d809-4213-8bcb-d9c54348c240_isrForCrosstalkSources_1029419_15.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0f014ff3-d809-4213-8bcb-d9c54348c240_isrForCrosstalkSources_1029419_15.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 548cfcec99b37198dd0c71e994c00476 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e3dd497f-6cd6-4a48-b610-fad86dfc2065_isrForCrosstalkSources_1029420_25.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e3dd497f-6cd6-4a48-b610-fad86dfc2065_isrForCrosstalkSources_1029420_25.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 848c2c2800130fbaa9b62c1cc619e656 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/aa685e13-e29d-4ef6-af6a-34442327e305_isrForCrosstalkSources_1029424_43.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/aa685e13-e29d-4ef6-af6a-34442327e305_isrForCrosstalkSources_1029424_43.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash ce682f7d2a427c472ee8a2bf41e2b650 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fab5be19-8c28-4fcf-a20a-74a690711c2b_isrForCrosstalkSources_1029418_3.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fab5be19-8c28-4fcf-a20a-74a690711c2b_isrForCrosstalkSources_1029418_3.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 61aa5b3b5a707c0d7859ebef067eb308 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d47c4c9e-f260-434a-b7ff-052ab5cff860_isrForCrosstalkSources_1029418_38.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d47c4c9e-f260-434a-b7ff-052ab5cff860_isrForCrosstalkSources_1029418_38.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 74718830c7de5a9ccd8a2a66d57407cb parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3747afb4-5115-4c05-9c14-795276a13ac1_isrForCrosstalkSources_1029418_41.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3747afb4-5115-4c05-9c14-795276a13ac1_isrForCrosstalkSources_1029418_41.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash d33eff052dd10abdbbca8652586a0ec2 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3368ab94-a1ca-4008-a3b2-619e55037aaf_isrForCrosstalkSources_1029420_29.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3368ab94-a1ca-4008-a3b2-619e55037aaf_isrForCrosstalkSources_1029420_29.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 67b62d12efbe3a2881df53f287dc6ca5 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/33ebdf42-bc11-4f2d-9d2c-71fc0ce07989_isrForCrosstalkSources_1029424_4.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/33ebdf42-bc11-4f2d-9d2c-71fc0ce07989_isrForCrosstalkSources_1029424_4.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash b3ec860d32527899d731f94283c0d33b parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dc5e256b-b0a0-48df-aa00-10fc90cf9bcf_isrForCrosstalkSources_1029420_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dc5e256b-b0a0-48df-aa00-10fc90cf9bcf_isrForCrosstalkSources_1029420_44.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 9b97e95fcb84c55cd2a7edab66655e0c parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/55800fa1-5983-4d4d-8865-c115750008a8_isrForCrosstalkSources_1029419_31.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/55800fa1-5983-4d4d-8865-c115750008a8_isrForCrosstalkSources_1029419_31.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 29f74cf483f34306c37b49216dd6128a parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/61d907db-b428-4138-ae22-41e50bd33952_isrForCrosstalkSources_1029418_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/61d907db-b428-4138-ae22-41e50bd33952_isrForCrosstalkSources_1029418_24.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash c8356102e584a668243f4df7359784e4 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dfcf82a1-39fe-4e6c-8d3e-c46eacd8b79c_isrForCrosstalkSources_1029424_52.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dfcf82a1-39fe-4e6c-8d3e-c46eacd8b79c_isrForCrosstalkSources_1029424_52.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 965e04282ebff7d103a3ea85daaaa194 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/80bfecde-4d22-4a03-886f-8cae45dc52fd_isrForCrosstalkSources_1029419_11.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/80bfecde-4d22-4a03-886f-8cae45dc52fd_isrForCrosstalkSources_1029419_11.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 030558ad3e787b5ca1db68411349e0a7 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e257b887-498d-4f74-a6e5-78daa3c48762_isrForCrosstalkSources_1029418_58.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e257b887-498d-4f74-a6e5-78daa3c48762_isrForCrosstalkSources_1029418_58.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 390632fd8fceff79a003f8aad798df38 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cc719f7a-3d30-47c9-a480-68247706bcbf_isrForCrosstalkSources_1029420_52.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cc719f7a-3d30-47c9-a480-68247706bcbf_isrForCrosstalkSources_1029420_52.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 27a3708b8c663ee2e6d5b62440130802 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b9f491cf-526a-4b92-a954-8f6abfde9e7c_isrForCrosstalkSources_1029425_5.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b9f491cf-526a-4b92-a954-8f6abfde9e7c_isrForCrosstalkSources_1029425_5.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash bc48a7ae03d79fc33e0b558eb4087b48 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/26355618-5c01-4fe8-a8c1-3fdad876e3f6_isrForCrosstalkSources_1029420_3.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/26355618-5c01-4fe8-a8c1-3fdad876e3f6_isrForCrosstalkSources_1029420_3.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 7a5aeb03796c10cb6df6c89cbc0eaf51 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/68555904-45c9-4672-a3af-df9f204e2f95_isrForCrosstalkSources_1029419_28.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/68555904-45c9-4672-a3af-df9f204e2f95_isrForCrosstalkSources_1029419_28.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash be297b785b2e41ad9f757d7dffbd7c0d parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/037eef55-556f-49fe-bd2d-6f614f64c357_isrForCrosstalkSources_1029426_27.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/037eef55-556f-49fe-bd2d-6f614f64c357_isrForCrosstalkSources_1029426_27.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 6286dd8514ecfecfd6bab1a5923b1944 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/6ee58579-0902-4c1e-b918-d73fcdd42a44_isrForCrosstalkSources_1029426_24.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/6ee58579-0902-4c1e-b918-d73fcdd42a44_isrForCrosstalkSources_1029426_24.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash a2cf9d8c52761db36d37091c8fb52986 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/1e93adec-3b71-4756-9b63-074b8913809a_isrForCrosstalkSources_1029426_44.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/1e93adec-3b71-4756-9b63-074b8913809a_isrForCrosstalkSources_1029426_44.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 28321915e937b8f9cb7529b26b796fed parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/073bf926-ae4d-41f0-a8f0-a63a7703f3d1_isrForCrosstalkSources_1029420_22.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/073bf926-ae4d-41f0-a8f0-a63a7703f3d1_isrForCrosstalkSources_1029420_22.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 0c0474b20d6a380f589945fab397eaa0 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2601c431-f69e-43d9-a51c-33679c5d2482_isrForCrosstalkSources_1029425_25.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2601c431-f69e-43d9-a51c-33679c5d2482_isrForCrosstalkSources_1029425_25.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 9a163adef8cad9e311ac99985ddac575 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/82949259-5063-42e2-936d-85393203b425_isrForCrosstalkSources_1029424_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/82949259-5063-42e2-936d-85393203b425_isrForCrosstalkSources_1029424_54.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash f69a36e51102ab85da78e00379161ee3 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/68148494-f6e6-4188-a674-59fce1f1d6fe_isrForCrosstalkSources_1029426_32.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/68148494-f6e6-4188-a674-59fce1f1d6fe_isrForCrosstalkSources_1029426_32.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 792d6aedd0aad1c96d6000d9d86daa66 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/0e76e83a-b9ef-4c14-9105-d47bdbb6cab5_isrForCrosstalkSources_1029425_59.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/0e76e83a-b9ef-4c14-9105-d47bdbb6cab5_isrForCrosstalkSources_1029425_59.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 6c94c3112066f5d41d958b8430a781fe parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/648cd1e8-f834-41ec-a088-208245634be0_isrForCrosstalkSources_1029426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/648cd1e8-f834-41ec-a088-208245634be0_isrForCrosstalkSources_1029426_53.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 59eaa32bb80c0cea50d7193dc5a94bd7 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0297a5c7-18f0-4d9c-81ec-5f44b6a9e682_isrForCrosstalkSources_1029424_47.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0297a5c7-18f0-4d9c-81ec-5f44b6a9e682_isrForCrosstalkSources_1029424_47.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash a3e659d1b106ec3b68106389caba47f4 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/817b68ed-3a48-48a4-9ed8-20f54421fe23_isrForCrosstalkSources_1029418_34.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/817b68ed-3a48-48a4-9ed8-20f54421fe23_isrForCrosstalkSources_1029418_34.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash ead592d4c4721ea8e961bb694c574f8d parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/048ec70e-102b-4faa-8ccd-03ecd443961c_isrForCrosstalkSources_1029425_3.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/048ec70e-102b-4faa-8ccd-03ecd443961c_isrForCrosstalkSources_1029425_3.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 5289f51c165439c00cbe93c106fa2bc2 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-16 11:08:30 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/acfd2285-2e66-4879-ab53-f583396d3f7c_isrForCrosstalkSources_1029418_59.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/acfd2285-2e66-4879-ab53-f583396d3f7c_isrForCrosstalkSources_1029418_59.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 36aae57f1329a99a22b5493d99ed138e parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc9377240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2278a3ff-07fe-47a6-86d6-b0c850771e3d_isrForCrosstalkSources_1029426_48.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2278a3ff-07fe-47a6-86d6-b0c850771e3d_isrForCrosstalkSources_1029426_48.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 31840cd4484c3e630e355797d6aae171 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/85da8d9e-35ec-4279-8ee1-beafd909278d_isrForCrosstalkSources_1029426_38.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/85da8d9e-35ec-4279-8ee1-beafd909278d_isrForCrosstalkSources_1029426_38.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash c453325a166cf53d1db34e4822b48d9f parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6f23d37d-9413-4c05-bf41-b6d43588db3b_isrForCrosstalkSources_1029420_27.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6f23d37d-9413-4c05-bf41-b6d43588db3b_isrForCrosstalkSources_1029420_27.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash ac52ea6203823c0238385739e66cebb3 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b5aef3ca-5da6-4bdf-be6c-a2c621e7ac67_isrForCrosstalkSources_1029424_56.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b5aef3ca-5da6-4bdf-be6c-a2c621e7ac67_isrForCrosstalkSources_1029424_56.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 9adb29fe732dc8b91df674e4eee7fdb1 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/045e2be6-e221-46cf-8c3d-650c888ac517_isrForCrosstalkSources_1029418_31.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/045e2be6-e221-46cf-8c3d-650c888ac517_isrForCrosstalkSources_1029418_31.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash e55167ac45b939143e230b21a8ae091a parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9caffb32-d24b-432a-bb42-88a533df3227_isrForCrosstalkSources_1029426_6.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9caffb32-d24b-432a-bb42-88a533df3227_isrForCrosstalkSources_1029426_6.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 6a720f000cfa4a649160b554b04f5fd0 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42dd7e60-410a-4a0c-875f-33da8abc2331_isrForCrosstalkSources_1029420_30.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42dd7e60-410a-4a0c-875f-33da8abc2331_isrForCrosstalkSources_1029420_30.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash b6d275e13cbeb67cf78ff9a562cc6196 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7c841969-a7e8-4951-b552-6aff6b4ab0fc_isrForCrosstalkSources_1029426_51.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7c841969-a7e8-4951-b552-6aff6b4ab0fc_isrForCrosstalkSources_1029426_51.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 9de0dfe264e766380c0cc05c5d16103e parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d280e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cc05929e-849c-4d5f-a1dd-ca7e5e23787e_isrForCrosstalkSources_1029425_32.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cc05929e-849c-4d5f-a1dd-ca7e5e23787e_isrForCrosstalkSources_1029425_32.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash d7cd7b85c459bda66b012009e335ab82 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/061a1c50-8d50-423d-991a-fe7d15e701c0_isrForCrosstalkSources_1029425_7.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/061a1c50-8d50-423d-991a-fe7d15e701c0_isrForCrosstalkSources_1029425_7.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 762e462e08f9eeef0cf00c377e3604b6 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 167 parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/ca366cb1-6786-452f-a6e2-c9a4cc8c6204_isrForCrosstalkSources_1029419_58.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/ca366cb1-6786-452f-a6e2-c9a4cc8c6204_isrForCrosstalkSources_1029419_58.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 6923415f953ff5573ceb5c1a951b0135 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9172c17f-ce74-471c-b8b9-6a2a160f52ae_isrForCrosstalkSources_1029425_19.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9172c17f-ce74-471c-b8b9-6a2a160f52ae_isrForCrosstalkSources_1029425_19.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 4530b5521624a912e626a62797bbd98c parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/239f6494-d6c1-4bf7-af1b-474f12684bf4_isrForCrosstalkSources_1029426_5.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/239f6494-d6c1-4bf7-af1b-474f12684bf4_isrForCrosstalkSources_1029426_5.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 2db9c768e3e6b52d185a8dae6a73d1ba parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c702fa64-05d6-455b-8494-ebc435ad48b1_isrForCrosstalkSources_1029418_6.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c702fa64-05d6-455b-8494-ebc435ad48b1_isrForCrosstalkSources_1029418_6.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 8ee720e49dd78a44737c28a54f02d4e4 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/956a6dee-d605-4218-8ed6-9ed9d348cae4_isrForCrosstalkSources_1029426_12.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/956a6dee-d605-4218-8ed6-9ed9d348cae4_isrForCrosstalkSources_1029426_12.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 5ff8db78199244ecf8970acace759f19 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8f5846cf-69df-4fdf-901e-f22b62c68cb9_isrForCrosstalkSources_1029424_34.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8f5846cf-69df-4fdf-901e-f22b62c68cb9_isrForCrosstalkSources_1029424_34.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 381e8d95e92a1e4bc5372b412a717a1d parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7ec2ddf7-8ea1-4287-9d24-083e1024477f_isrForCrosstalkSources_1029425_24.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7ec2ddf7-8ea1-4287-9d24-083e1024477f_isrForCrosstalkSources_1029425_24.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 885d2fda9d883add05a258a0f21f92fb parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d285e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 167 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 41 more blocks parsl.executors.status_handling INFO: Scaling out by 41 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-16 11:08:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-16 11:08:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 11:08:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd92ce92-5628-4a9f-8d88-e7e8526eaeb4_isrForCrosstalkSources_1029424_61.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd92ce92-5628-4a9f-8d88-e7e8526eaeb4_isrForCrosstalkSources_1029424_61.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 644b8d5b27c493d9b5025749a574b6c9 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1d930e6d-db74-4ba8-b667-ba527dfd2ea0_isrForCrosstalkSources_1029418_25.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1d930e6d-db74-4ba8-b667-ba527dfd2ea0_isrForCrosstalkSources_1029418_25.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 23f4a7c801215efa160aaefbd7f15b78 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/71250836-353d-4a6f-8d63-959668de4059_isrForCrosstalkSources_1029424_29.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/71250836-353d-4a6f-8d63-959668de4059_isrForCrosstalkSources_1029424_29.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 4859485edc325ed4003ae3637c3c4421 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/21d934c9-d4fb-40de-8508-d2581c3274f7_isrForCrosstalkSources_1029426_56.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/21d934c9-d4fb-40de-8508-d2581c3274f7_isrForCrosstalkSources_1029426_56.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash c84697b883e29dec202d4c95feaadc32 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d289a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c66f8c10-73f9-4825-a674-2ecb23d34aa3_isrForCrosstalkSources_1029420_19.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c66f8c10-73f9-4825-a674-2ecb23d34aa3_isrForCrosstalkSources_1029420_19.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash a65f0e95602586c943b5677805540fbd parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/85df1d50-2b3b-4e03-ab67-4e3fb67c2bc7_isrForCrosstalkSources_1029425_16.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/85df1d50-2b3b-4e03-ab67-4e3fb67c2bc7_isrForCrosstalkSources_1029425_16.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 3d7765a7c6a73575b91821453d5e4ee5 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac0dac1-a00c-4176-9345-f268247b9032_isrForCrosstalkSources_1029424_50.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac0dac1-a00c-4176-9345-f268247b9032_isrForCrosstalkSources_1029424_50.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 2c7e0976da7f325aa84d32edb3abd52c parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d287c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/031c17ce-d930-4bb6-a3a8-057e7a732654_isrForCrosstalkSources_1029419_49.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/031c17ce-d930-4bb6-a3a8-057e7a732654_isrForCrosstalkSources_1029419_49.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash ea86d1c89baeae322eb3c24c0f6c1043 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/278a84f1-a260-4e6b-a174-3907dc3ad8a7_isrForCrosstalkSources_1029420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/278a84f1-a260-4e6b-a174-3907dc3ad8a7_isrForCrosstalkSources_1029420_7.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 5991d6ff160ec49c532cb444b78b929e parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ccbb8877-c50f-45b2-9eb8-733f7aacbe50_isrForCrosstalkSources_1029426_8.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ccbb8877-c50f-45b2-9eb8-733f7aacbe50_isrForCrosstalkSources_1029426_8.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 637ee98457d4553db55c3cad54ae861c parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/612c07a2-87a9-4c67-8194-88bd82e370c5_isrForCrosstalkSources_1029418_20.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/612c07a2-87a9-4c67-8194-88bd82e370c5_isrForCrosstalkSources_1029418_20.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash bfa1223f31e1b5a2a6a572ae3345d668 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3561db0-2e34-4c02-8c8d-3020f796b3e2_isrForCrosstalkSources_1029419_7.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3561db0-2e34-4c02-8c8d-3020f796b3e2_isrForCrosstalkSources_1029419_7.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 7d3ecee5e92b398e94d1cc7c2a8b0689 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/2ebbab5a-5630-41f7-8764-0ef990dde462_isrForCrosstalkSources_1029420_4.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/2ebbab5a-5630-41f7-8764-0ef990dde462_isrForCrosstalkSources_1029420_4.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 38281ef56c5f3fa6c16bae54eedd6799 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4546d368-8f6c-4e02-b445-589ed5f46409_isrForCrosstalkSources_1029425_57.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4546d368-8f6c-4e02-b445-589ed5f46409_isrForCrosstalkSources_1029425_57.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash bdc659892bcc21771ce9f6e0d1ca6ba8 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b4f7c228-038f-401e-8dd4-5eb928088b10_isrForCrosstalkSources_1029419_45.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b4f7c228-038f-401e-8dd4-5eb928088b10_isrForCrosstalkSources_1029419_45.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash d2873c176a8f7bca3eebebbb598047e8 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/55536c30-7ed5-4dc8-a086-12a2517edddb_isrForCrosstalkSources_1029418_51.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/55536c30-7ed5-4dc8-a086-12a2517edddb_isrForCrosstalkSources_1029418_51.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 6ba4683bf22efa0bed212b6874395ef1 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c62c15da-a8ba-4ba5-aa92-04d5c3cd2ed8_isrForCrosstalkSources_1029425_21.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c62c15da-a8ba-4ba5-aa92-04d5c3cd2ed8_isrForCrosstalkSources_1029425_21.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 62fd06a98ad31c62425ff9a7ee02220c parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d291c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fe70c1ae-db99-4bda-a938-36c531aa0533_isrForCrosstalkSources_1029424_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fe70c1ae-db99-4bda-a938-36c531aa0533_isrForCrosstalkSources_1029424_27.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash fb40940c3c79a011d9b22392be300e4f parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10e5474b-ed65-4c46-9250-104410c50210_isrForCrosstalkSources_1029425_38.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10e5474b-ed65-4c46-9250-104410c50210_isrForCrosstalkSources_1029425_38.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash af94c09e81bd488a80649c2327781cc4 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8ba459a0-6e93-4b13-8424-a84bb7c20394_isrForCrosstalkSources_1029425_62.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8ba459a0-6e93-4b13-8424-a84bb7c20394_isrForCrosstalkSources_1029425_62.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash eb0892f0387c70b0558577e395ca341f parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d294e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/446633f8-364a-4925-8f22-3fd6f1654b1e_isrForCrosstalkSources_1029425_11.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/446633f8-364a-4925-8f22-3fd6f1654b1e_isrForCrosstalkSources_1029425_11.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 80144870f00404abba796855b8fe7599 parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c9dec7e7-6afc-4785-893d-1e768604affe_isrForCrosstalkSources_1029424_23.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c9dec7e7-6afc-4785-893d-1e768604affe_isrForCrosstalkSources_1029424_23.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 6038f43024e6d1a87a751bf82d2220da parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d293a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3983dce3-7641-4361-a3a6-07e85fca594a_isrForCrosstalkSources_1029426_9.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3983dce3-7641-4361-a3a6-07e85fca594a_isrForCrosstalkSources_1029426_9.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 3901cb38ff2a75902194aa4cf4bc5b2e parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/426068b7-49e6-44f5-a496-d891fa7b14b3_isrForCrosstalkSources_1029426_7.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/426068b7-49e6-44f5-a496-d891fa7b14b3_isrForCrosstalkSources_1029426_7.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 8445525a0a21b63a96c50e0327306520 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d1c226fc-46fd-4921-a202-f4b71891db31_isrForCrosstalkSources_1029418_55.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d1c226fc-46fd-4921-a202-f4b71891db31_isrForCrosstalkSources_1029418_55.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash d31e58cdbcea1152e5fc7951e01c5d02 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd186c20-d5ee-4873-8260-57d91cf890ab_isrForCrosstalkSources_1029424_62.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd186c20-d5ee-4873-8260-57d91cf890ab_isrForCrosstalkSources_1029424_62.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 87f9ceb7c8c38d8fc7bf2d120c39aa56 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d296c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a62aede1-3720-4703-af11-b406b3732432_isrForCrosstalkSources_1029425_52.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a62aede1-3720-4703-af11-b406b3732432_isrForCrosstalkSources_1029425_52.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash 0a699df83657fa9cdd6d2497d6709cac parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d298a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/150515ed-2d44-49a4-b6e3-e0d80a24d751_isrForCrosstalkSources_1029420_46.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/150515ed-2d44-49a4-b6e3-e0d80a24d751_isrForCrosstalkSources_1029420_46.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 529b7e5516668dffeb668822c046e457 parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/061c68bb-7226-49c2-8645-1e29d6c2a671_isrForCrosstalkSources_1029424_30.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/061c68bb-7226-49c2-8645-1e29d6c2a671_isrForCrosstalkSources_1029424_30.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 0d8b22b9cf9d0935db11944acb5218d6 parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42339ffc-c0b5-4322-a09c-2e8c44fc6373_isrForCrosstalkSources_1029420_47.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42339ffc-c0b5-4322-a09c-2e8c44fc6373_isrForCrosstalkSources_1029420_47.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 94392ac04bb9f1d77ddaeece2f3f82eb parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e6da8f24-3c51-4909-97e0-06da2cca8c19_isrForCrosstalkSources_1029419_9.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e6da8f24-3c51-4909-97e0-06da2cca8c19_isrForCrosstalkSources_1029419_9.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash db5e1fc28aecc0d39029e16a9fc8e145 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/70387920-83e9-421e-ac50-2e06f9c44390_isrForCrosstalkSources_1029425_17.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/70387920-83e9-421e-ac50-2e06f9c44390_isrForCrosstalkSources_1029425_17.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 9a3c73d767d5224e314861de5b39b3ac parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1647661b-b141-488e-ac88-53b02e10d028_isrForCrosstalkSources_1029424_40.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1647661b-b141-488e-ac88-53b02e10d028_isrForCrosstalkSources_1029424_40.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 5fbd503fe6fe3d6d0508dd4f3d979f24 parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d299e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/83b2d31f-ba00-42b9-b75f-624fe7c87ec0_isrForCrosstalkSources_1029426_43.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/83b2d31f-ba00-42b9-b75f-624fe7c87ec0_isrForCrosstalkSources_1029426_43.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash e8ba6063be57f9fcaf2b2f95877695d0 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/dc0825fc-b06b-45ce-94b8-0daf36d5260c_isrForCrosstalkSources_1029419_36.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/dc0825fc-b06b-45ce-94b8-0daf36d5260c_isrForCrosstalkSources_1029419_36.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash cb1246d8062d8e67076978b1b180a1b4 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7f06e93f-4a46-4932-9ae9-0ef9f4239b08_isrForCrosstalkSources_1029425_29.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7f06e93f-4a46-4932-9ae9-0ef9f4239b08_isrForCrosstalkSources_1029425_29.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash cc950a34ef897926d85ea4b8be02adb9 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/9abcfcf6-e70d-49db-83aa-1ca1a0c575ca_isrForCrosstalkSources_1029420_61.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/9abcfcf6-e70d-49db-83aa-1ca1a0c575ca_isrForCrosstalkSources_1029420_61.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash d6d5d2444a305bb3c44eea2707332c2f parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e2dc0924-7498-4be5-8bb1-de2b9c630d68_isrForCrosstalkSources_1029419_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e2dc0924-7498-4be5-8bb1-de2b9c630d68_isrForCrosstalkSources_1029419_4.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 6c40ebc77e2674ddda8aac435ad3babb parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/54ce4dff-7aa4-433b-9904-013bfd1742ee_isrForCrosstalkSources_1029420_58.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/54ce4dff-7aa4-433b-9904-013bfd1742ee_isrForCrosstalkSources_1029420_58.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash c1294285c52311e89920ceddbfe10501 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/263345d7-a660-491b-a9ea-5dd83e6d6bfb_isrForCrosstalkSources_1029419_27.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/263345d7-a660-491b-a9ea-5dd83e6d6bfb_isrForCrosstalkSources_1029419_27.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash aeb44f936d258b0ef128fa647707a0a5 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8fdcb722-413d-4943-b0d1-8892892cb2a4_isrForCrosstalkSources_1029425_54.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8fdcb722-413d-4943-b0d1-8892892cb2a4_isrForCrosstalkSources_1029425_54.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash fa9a9d1be621d1fb1eeeb86fa12ad9d6 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/55fc125b-a07c-42ba-961c-f9a26d246efd_isrForCrosstalkSources_1029424_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/55fc125b-a07c-42ba-961c-f9a26d246efd_isrForCrosstalkSources_1029424_46.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 642b6197eaeceec15d91c0d6f01dc62f parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/490da1ea-4bfb-4545-a009-25298ac38aa9_isrForCrosstalkSources_1029426_55.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/490da1ea-4bfb-4545-a009-25298ac38aa9_isrForCrosstalkSources_1029426_55.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash e33739a9deee970d18c52fb41a024288 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02b17ef7-1221-4edb-913f-f07c62cd260f_isrForCrosstalkSources_1029419_16.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02b17ef7-1221-4edb-913f-f07c62cd260f_isrForCrosstalkSources_1029419_16.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 541c38367b86577be1add94f9f2f2bb2 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3dacfb96-a670-4099-8c4a-3c4d36250014_isrForCrosstalkSources_1029426_39.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3dacfb96-a670-4099-8c4a-3c4d36250014_isrForCrosstalkSources_1029426_39.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 95526e021316d1fa007a30b8b8e3d9e8 parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c233a2a0-3a40-48f6-87f4-5167386e2511_isrForCrosstalkSources_1029419_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c233a2a0-3a40-48f6-87f4-5167386e2511_isrForCrosstalkSources_1029419_17.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 2b1baa0bffabaa92cafe85a6fda99a52 parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c4f6327f-0a61-4c0b-ae9e-dcb189f43468_isrForCrosstalkSources_1029424_22.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c4f6327f-0a61-4c0b-ae9e-dcb189f43468_isrForCrosstalkSources_1029424_22.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash bb696bd7136597d1f3ae0cbb1471bb3b parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d864c404-963c-412a-a120-b4b1bce9236e_isrForCrosstalkSources_1029425_4.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d864c404-963c-412a-a120-b4b1bce9236e_isrForCrosstalkSources_1029425_4.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 9b4156d094836ca46aa8f6eacd4ecf48 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/aad1db82-540d-4cb8-8024-03f365588f0c_isrForCrosstalkSources_1029419_8.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/aad1db82-540d-4cb8-8024-03f365588f0c_isrForCrosstalkSources_1029419_8.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash e79cc7b42662a144567a7b7ff4fc92ad parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d46fa904-816d-4aaf-b857-f1386b35df49_isrForCrosstalkSources_1029419_56.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d46fa904-816d-4aaf-b857-f1386b35df49_isrForCrosstalkSources_1029419_56.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 66b095a4fa66eb293d0d1b654e7916ba parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fb8d5e0b-0701-4ea6-901a-298bb8249f96_isrForCrosstalkSources_1029424_16.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fb8d5e0b-0701-4ea6-901a-298bb8249f96_isrForCrosstalkSources_1029424_16.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 71367c206cc75e2f4e335dd8cd2e9f2e parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b8e14230-3c5a-4a58-8676-003cdeea3a21_isrForCrosstalkSources_1029418_39.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b8e14230-3c5a-4a58-8676-003cdeea3a21_isrForCrosstalkSources_1029418_39.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash bf41b1eef507d07096043293c535c600 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35d78006-e1c7-4264-bb73-c8e1b06bf1e1_isrForCrosstalkSources_1029425_60.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35d78006-e1c7-4264-bb73-c8e1b06bf1e1_isrForCrosstalkSources_1029425_60.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash a1b967cde814224082aec1fe2278e7f1 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/50eb7eec-980f-4742-9ef6-93a7c8163af5_isrForCrosstalkSources_1029426_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/50eb7eec-980f-4742-9ef6-93a7c8163af5_isrForCrosstalkSources_1029426_40.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 0bd67bbacc30a6675797c8285ea3e2b6 parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/279f827d-f34a-4bec-a8b3-637836d17db1_isrForCrosstalkSources_1029418_33.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/279f827d-f34a-4bec-a8b3-637836d17db1_isrForCrosstalkSources_1029418_33.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash 072d17563f2ce6c7b34ff77018d4123b parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60ef00ac-658d-4846-b25b-80cb4ea06ad9_isrForCrosstalkSources_1029420_1.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60ef00ac-658d-4846-b25b-80cb4ea06ad9_isrForCrosstalkSources_1029420_1.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash c8384586d5a1716c2f3d58c58009e9e5 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f11b13c5-5795-4daf-a745-7b51d19da771_isrForCrosstalkSources_1029418_56.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f11b13c5-5795-4daf-a745-7b51d19da771_isrForCrosstalkSources_1029418_56.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 2fd1865e669e517d8c5122b930a6bd94 parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/1a3b5786-47e6-491e-ac22-861824b5f436_isrForCrosstalkSources_1029419_37.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/1a3b5786-47e6-491e-ac22-861824b5f436_isrForCrosstalkSources_1029419_37.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash a05f3acf63bcc4f6ad6c9b914682387a parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bf9d60fa-1590-4120-bc76-f98a0df87402_isrForCrosstalkSources_1029418_52.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bf9d60fa-1590-4120-bc76-f98a0df87402_isrForCrosstalkSources_1029418_52.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 3d4476e0f037505ed9c5b2fc6a11a7f4 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/53ff6062-cbb6-45b0-9754-e89f165357a0_isrForCrosstalkSources_1029419_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/53ff6062-cbb6-45b0-9754-e89f165357a0_isrForCrosstalkSources_1029419_21.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash c8437d4d4d194e039e7f7dad4e44d827 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/bf2f08e4-7ed7-4d0b-8ad4-9a00e9acd4b0_isrForCrosstalkSources_1029425_13.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/bf2f08e4-7ed7-4d0b-8ad4-9a00e9acd4b0_isrForCrosstalkSources_1029425_13.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 6b80d7dad616dc8736fe93e32bbd78c0 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dd93becd-211d-48e3-940d-e05f2ed9adea_isrForCrosstalkSources_1029420_24.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dd93becd-211d-48e3-940d-e05f2ed9adea_isrForCrosstalkSources_1029420_24.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 3df1272e8f6b28baca995dafaca5ce5d parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c8b510c9-ebee-44a1-a02a-b7a476f04005_isrForCrosstalkSources_1029425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c8b510c9-ebee-44a1-a02a-b7a476f04005_isrForCrosstalkSources_1029425_51.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 493db03fa3b7b3269bfdf560c4ebc80c parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac744d3-f3f0-4a2d-a1d0-f94a4b3923bc_isrForCrosstalkSources_1029424_6.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac744d3-f3f0-4a2d-a1d0-f94a4b3923bc_isrForCrosstalkSources_1029424_6.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash cb95f95c28bbe90bafbacbd01a9a5d3c parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ddb0f8d4-4c41-4c6e-bfb2-21b7421e39a3_isrForCrosstalkSources_1029424_26.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ddb0f8d4-4c41-4c6e-bfb2-21b7421e39a3_isrForCrosstalkSources_1029424_26.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 2600672ea6125028bac0048b8ba548d3 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/aeabf46f-d3a1-4a3f-926b-745f3d2ccb2d_isrForCrosstalkSources_1029425_45.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/aeabf46f-d3a1-4a3f-926b-745f3d2ccb2d_isrForCrosstalkSources_1029425_45.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 87d069dcafb61c0e8c4788eb4ca5b46f parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4bcdcd53-fd71-494b-875e-5ac3b56b52ec_isrForCrosstalkSources_1029426_41.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4bcdcd53-fd71-494b-875e-5ac3b56b52ec_isrForCrosstalkSources_1029426_41.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 3183b4cb6a6bba99e529722632a215a3 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/5c7ebf9f-2ee4-4fb3-b549-817ad23cefa8_isrForCrosstalkSources_1029418_53.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/5c7ebf9f-2ee4-4fb3-b549-817ad23cefa8_isrForCrosstalkSources_1029418_53.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash 145163953fd635ef31e2d7a4c9a1f016 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/830e9542-fa9c-483f-882f-8cd3af576b44_isrForCrosstalkSources_1029419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/830e9542-fa9c-483f-882f-8cd3af576b44_isrForCrosstalkSources_1029419_19.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash 9739c6b3fa76c27f3e2efba4106a9eda parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1221ca07-df5b-485b-9d4b-5eb018ec7281_isrForCrosstalkSources_1029418_42.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1221ca07-df5b-485b-9d4b-5eb018ec7281_isrForCrosstalkSources_1029418_42.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 7e205c7082ed227a2f32238d9f8cdc0f parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0be4b0a0-688e-486b-b04c-10d84c1e4216_isrForCrosstalkSources_1029424_25.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0be4b0a0-688e-486b-b04c-10d84c1e4216_isrForCrosstalkSources_1029424_25.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 8af9b47aff58b41528acadcf55ff7e1e parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fc228add-e7bc-42a0-9893-7b2cb38fa033_isrForCrosstalkSources_1029420_48.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fc228add-e7bc-42a0-9893-7b2cb38fa033_isrForCrosstalkSources_1029420_48.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 40b7daadd237e5679971ddb37eec1e3f parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd38674-80aa-4469-9e06-335d2bdec9f1_isrForCrosstalkSources_1029420_50.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd38674-80aa-4469-9e06-335d2bdec9f1_isrForCrosstalkSources_1029420_50.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 2a8e994759d0a2eecc9165116dbd3821 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87d0674e-939d-4a75-9fe0-94170a5091c8_isrForCrosstalkSources_1029424_10.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87d0674e-939d-4a75-9fe0-94170a5091c8_isrForCrosstalkSources_1029424_10.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 107a71d9571af56c2df006de70198853 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/33601bb5-d4ce-40ea-bc97-b901a15f2637_isrForCrosstalkSources_1029419_42.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/33601bb5-d4ce-40ea-bc97-b901a15f2637_isrForCrosstalkSources_1029419_42.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash efddd838289f1dd9b154a36343d57ab9 parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/431f0b42-4701-425e-a8f6-606cb3b74296_isrForCrosstalkSources_1029426_23.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/431f0b42-4701-425e-a8f6-606cb3b74296_isrForCrosstalkSources_1029426_23.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 018a07923e51be1fcfa717dde0a467c0 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f19ab51b-02fc-4a37-a9b5-1fedc9268ba6_isrForCrosstalkSources_1029426_45.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f19ab51b-02fc-4a37-a9b5-1fedc9268ba6_isrForCrosstalkSources_1029426_45.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 6752b8aebdfc37bc92100ad8f60fa7b7 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3fdaba97-bb82-4f18-ba2b-42b422d76cca_isrForCrosstalkSources_1029420_59.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3fdaba97-bb82-4f18-ba2b-42b422d76cca_isrForCrosstalkSources_1029420_59.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 38e12d4106b6a4c43eea719e2abe5bb2 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/345aeb0d-be8a-4109-9b07-58b84962054e_isrForCrosstalkSources_1029419_55.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/345aeb0d-be8a-4109-9b07-58b84962054e_isrForCrosstalkSources_1029419_55.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash bbaf62acf291f3767608f65e8104c42d parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/41b88e24-b592-4f30-87bd-9c5fc8f68419_isrForCrosstalkSources_1029425_61.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/41b88e24-b592-4f30-87bd-9c5fc8f68419_isrForCrosstalkSources_1029425_61.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 6aa2d0a8a5ccb9dcb993ac1093b9c154 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fce9c334-d63b-4b76-99c5-ff0b1db0e9bb_isrForCrosstalkSources_1029418_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fce9c334-d63b-4b76-99c5-ff0b1db0e9bb_isrForCrosstalkSources_1029418_4.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 2480f8f9847b3bb11e66ef685802bb04 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d29ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b29e5666-920a-4ab9-9b67-cbdcc87fcfba_isrForCrosstalkSources_1029418_35.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b29e5666-920a-4ab9-9b67-cbdcc87fcfba_isrForCrosstalkSources_1029418_35.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 56779056dad402dfceab0596e6ecdb2c parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/41057a1e-70b6-45c4-bd04-76f40cb36f81_isrForCrosstalkSources_1029420_12.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/41057a1e-70b6-45c4-bd04-76f40cb36f81_isrForCrosstalkSources_1029420_12.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 972d4011fdb29c74e195d10fe2b875a5 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/695dd3f4-757e-4367-8ff8-8b0d35633a70_isrForCrosstalkSources_1029425_53.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/695dd3f4-757e-4367-8ff8-8b0d35633a70_isrForCrosstalkSources_1029425_53.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash 9b9a02857bb2934c24dce340c17602c9 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d2a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8b10307b-ecc5-4394-bd34-d3c96e2cf906_isrForCrosstalkSources_1029425_14.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8b10307b-ecc5-4394-bd34-d3c96e2cf906_isrForCrosstalkSources_1029425_14.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash c830f5ade6510c515ffca6c790b5fd75 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/bc6f4b85-9887-4572-a413-0085a248120d_isrForCrosstalkSources_1029426_50.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/bc6f4b85-9887-4572-a413-0085a248120d_isrForCrosstalkSources_1029426_50.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 00667634f3c352210aca5fcad960f48d parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6a740928-c18a-4902-a6f5-fc5facc4dda3_isrForCrosstalkSources_1029420_11.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6a740928-c18a-4902-a6f5-fc5facc4dda3_isrForCrosstalkSources_1029420_11.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 7ad092c96e8e80294dfc92105c686dc0 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/822ad98a-f39f-40d8-a17a-c27b14fbaeef_isrForCrosstalkSources_1029425_55.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/822ad98a-f39f-40d8-a17a-c27b14fbaeef_isrForCrosstalkSources_1029425_55.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 222059865bd56a18fdfe838c6a8c1d12 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d8bd2247-161b-421d-b7ea-78fdbf7084b1_isrForCrosstalkSources_1029425_58.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d8bd2247-161b-421d-b7ea-78fdbf7084b1_isrForCrosstalkSources_1029425_58.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash 6a1e2a2dc2654e44e82d6f413b767a9f parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ba3f1610-3edb-47dc-b175-f800978e3ec5_isrForCrosstalkSources_1029426_57.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ba3f1610-3edb-47dc-b175-f800978e3ec5_isrForCrosstalkSources_1029426_57.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash f8527d511fe3e8a95da1d244af2f74b7 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/fb233a35-8f65-4d39-8b6c-f3f84c59fbd2_isrForCrosstalkSources_1029426_58.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/fb233a35-8f65-4d39-8b6c-f3f84c59fbd2_isrForCrosstalkSources_1029426_58.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 82f6866ba56e36c9fdf6665c0afff9d3 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/78fdc670-4a80-45b3-915b-4d88bbf49026_isrForCrosstalkSources_1029420_14.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/78fdc670-4a80-45b3-915b-4d88bbf49026_isrForCrosstalkSources_1029420_14.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 4bbbddddaa04485aba9c6e8e6121e6be parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c9135493-0aa5-4c9e-8ea0-94704babf0bd_isrForCrosstalkSources_1029426_36.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c9135493-0aa5-4c9e-8ea0-94704babf0bd_isrForCrosstalkSources_1029426_36.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash 7f4fa515d341b11a903fa4ec59c21fb3 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e7761761-226a-4f85-80d5-05aadc403180_isrForCrosstalkSources_1029426_28.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e7761761-226a-4f85-80d5-05aadc403180_isrForCrosstalkSources_1029426_28.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 99dcfba50d3a9c6d005db76dc9a18c00 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/22d257ef-8056-4262-b13a-554c02e62aca_isrForCrosstalkSources_1029426_26.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/22d257ef-8056-4262-b13a-554c02e62aca_isrForCrosstalkSources_1029426_26.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash a60eb74e7a7737c1ebc540e0b4d6b955 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/08c87446-46d8-4403-a458-e0961bd92dd6_isrForCrosstalkSources_1029420_38.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/08c87446-46d8-4403-a458-e0961bd92dd6_isrForCrosstalkSources_1029420_38.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash b5082366e492ea04026702e44ec25398 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/f216d3ca-1f19-4c89-bf9d-be2d4dbd8971_isrForCrosstalkSources_1029424_11.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/f216d3ca-1f19-4c89-bf9d-be2d4dbd8971_isrForCrosstalkSources_1029424_11.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash 6559efb8859bf582958bb38ed4b06d0b parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ad784513-f34d-49f0-b7b8-0e73d85d9657_isrForCrosstalkSources_1029418_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ad784513-f34d-49f0-b7b8-0e73d85d9657_isrForCrosstalkSources_1029418_27.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 8a4256fc9d3f2e588a2339b93f8022fa parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/05897549-661a-4c9f-a5fe-7a3360b6c111_isrForCrosstalkSources_1029426_33.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/05897549-661a-4c9f-a5fe-7a3360b6c111_isrForCrosstalkSources_1029426_33.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 4a6a39617e4467edc29d58b6a06b50d3 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/edd99368-51e1-42a1-a22b-28ff57d9d8c8_isrForCrosstalkSources_1029419_41.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/edd99368-51e1-42a1-a22b-28ff57d9d8c8_isrForCrosstalkSources_1029419_41.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash b0e4a558322825f1105cf968a9c1a4bb parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-16 11:08:31 proc_lsst.multi:170 [INFO] [multi] job_id 17113718 proc_lsst.multi INFO: [multi] job_id 17113718 2024-03-16 11:08:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113718 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113718 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17113718 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-16 11:08:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-16 11:08:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2251dce1-5b57-4b38-ad65-a9073308b444_isrForCrosstalkSources_1029418_9.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2251dce1-5b57-4b38-ad65-a9073308b444_isrForCrosstalkSources_1029418_9.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash 77b5866e79220202634e1bf9385e891f parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a838bd8b-e61e-4e86-b753-d079f79a39b0_isrForCrosstalkSources_1029420_5.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a838bd8b-e61e-4e86-b753-d079f79a39b0_isrForCrosstalkSources_1029420_5.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 6d83f5448a9ee75245fdc894e45b30ce parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ea010178-4eac-41da-aa8b-ee9c383c7706_isrForCrosstalkSources_1029424_44.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ea010178-4eac-41da-aa8b-ee9c383c7706_isrForCrosstalkSources_1029424_44.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash f28ac833cf41603949d9851dff7e167f parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3d28c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d8f4c6b6-8820-4566-ab66-86e422f8daac_isrForCrosstalkSources_1029419_54.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d8f4c6b6-8820-4566-ab66-86e422f8daac_isrForCrosstalkSources_1029419_54.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 22684ad6245d8c2f123f162756ec0fe9 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7d2cf6f5-aaab-48a5-8e0f-5fe52ec910d2_isrForCrosstalkSources_1029425_10.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7d2cf6f5-aaab-48a5-8e0f-5fe52ec910d2_isrForCrosstalkSources_1029425_10.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 3faed67135bc0c2205e35503c62b1c8b parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/5a0b355d-40ed-4928-ae59-8697afb1ccc3_isrForCrosstalkSources_1029419_48.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/5a0b355d-40ed-4928-ae59-8697afb1ccc3_isrForCrosstalkSources_1029419_48.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 5d937f3f55c688f3f916b8e8a70ace0a parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/73720cef-851a-482e-9ef7-7a8bd5321c90_isrForCrosstalkSources_1029419_10.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/73720cef-851a-482e-9ef7-7a8bd5321c90_isrForCrosstalkSources_1029419_10.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash dd41c8dbc1a0d3e34b05be335ed5b496 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e061b789-e4b9-4e78-964a-bae20512b22f_isrForCrosstalkSources_1029420_54.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e061b789-e4b9-4e78-964a-bae20512b22f_isrForCrosstalkSources_1029420_54.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash 8564571fd15ffc501e74a68e89533ed1 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4baeae98-2c4c-4cc0-a5cd-140dccdb8816_isrForCrosstalkSources_1029426_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4baeae98-2c4c-4cc0-a5cd-140dccdb8816_isrForCrosstalkSources_1029426_59.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash a4455802bcd03cc9a49198dfe7a727b3 parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd679752-11ce-48f8-bbd9-ca534d4c9633_isrForCrosstalkSources_1029425_44.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd679752-11ce-48f8-bbd9-ca534d4c9633_isrForCrosstalkSources_1029425_44.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash af02eceae962c6922a80ab081fc4dff3 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/96f70810-b041-437b-a9dc-2e344c482a38_isrForCrosstalkSources_1029424_5.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/96f70810-b041-437b-a9dc-2e344c482a38_isrForCrosstalkSources_1029424_5.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash dfcd4611bd78c54ebd5b3cfd56e189f1 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/55bb8c74-8d95-426a-90c5-85d05b6daad3_isrForCrosstalkSources_1029420_2.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/55bb8c74-8d95-426a-90c5-85d05b6daad3_isrForCrosstalkSources_1029420_2.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 596183c1e7c2bf84ae11fef209d998d3 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c82eb9ce-ce26-419f-9e44-bd85a7dd490f_isrForCrosstalkSources_1029420_20.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c82eb9ce-ce26-419f-9e44-bd85a7dd490f_isrForCrosstalkSources_1029420_20.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 9b7f8727e505f975f7ee8b546a1afb10 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/12885372-057b-4d68-bd84-e2838f1592f8_isrForCrosstalkSources_1029426_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/12885372-057b-4d68-bd84-e2838f1592f8_isrForCrosstalkSources_1029426_29.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 9800675fa27af2aff98aa88a4364d261 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/66c1f960-12fb-44e7-ae46-2155cf85def0_isrForCrosstalkSources_1029426_19.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/66c1f960-12fb-44e7-ae46-2155cf85def0_isrForCrosstalkSources_1029426_19.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 22db6e0d0b01b43f419d7ad749ab259a parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbcea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/618911b2-6a47-49b8-a72d-8933ed96906b_isrForCrosstalkSources_1029420_15.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/618911b2-6a47-49b8-a72d-8933ed96906b_isrForCrosstalkSources_1029420_15.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 717457c343c16df491409b0438d6e552 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3104852-068d-4c0d-9774-89ca246cd5be_isrForCrosstalkSources_1029419_60.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3104852-068d-4c0d-9774-89ca246cd5be_isrForCrosstalkSources_1029419_60.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 6a74e1a26178c72d25fc4b78678e443c parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/05317ca0-01bc-4268-a5e8-fc2011528316_isrForCrosstalkSources_1029425_49.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/05317ca0-01bc-4268-a5e8-fc2011528316_isrForCrosstalkSources_1029425_49.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 310bb8c4bd25e5f35150d6421fa4988f parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cf2a4865-d6a3-49b9-9bee-460d2605b05e_isrForCrosstalkSources_1029425_8.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cf2a4865-d6a3-49b9-9bee-460d2605b05e_isrForCrosstalkSources_1029425_8.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash e6aef2f6ffe7faf6430389634bcb7bad parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cec7f866-b785-4101-b075-d72537188622_isrForCrosstalkSources_1029420_36.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cec7f866-b785-4101-b075-d72537188622_isrForCrosstalkSources_1029420_36.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash 5ed6db41907f70d00946784842ee8d27 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/38f38998-235f-4351-884d-6a41e29d83cb_isrForCrosstalkSources_1029418_14.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/38f38998-235f-4351-884d-6a41e29d83cb_isrForCrosstalkSources_1029418_14.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 22d88f14013483c00f27cccdf45ee45a parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/66fd287b-ca8e-4c4e-931b-e2acdced2bd9_isrForCrosstalkSources_1029418_49.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/66fd287b-ca8e-4c4e-931b-e2acdced2bd9_isrForCrosstalkSources_1029418_49.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash f70bbd280eaed7c44835ea65d9fc8aeb parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6d390c1d-3979-496d-848a-bbc59bda5aef_isrForCrosstalkSources_1029420_41.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6d390c1d-3979-496d-848a-bbc59bda5aef_isrForCrosstalkSources_1029420_41.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 5e361ac2d3dc3d1cd58cf52198639bf6 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b0d27486-d5dd-423f-aa95-25002ad51c51_isrForCrosstalkSources_1029425_35.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b0d27486-d5dd-423f-aa95-25002ad51c51_isrForCrosstalkSources_1029425_35.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 8d383bf7e93289e804c4ec03fb106fd7 parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3645be2f-be3a-4a46-a481-206445b9fc3d_isrForCrosstalkSources_1029418_60.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3645be2f-be3a-4a46-a481-206445b9fc3d_isrForCrosstalkSources_1029418_60.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 6403fe06d9583fd82ad86996c7b1ea86 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ba06b8a8-e8b2-4bdd-9935-5086a820176d_isrForCrosstalkSources_1029418_32.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ba06b8a8-e8b2-4bdd-9935-5086a820176d_isrForCrosstalkSources_1029418_32.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash c1fdb94dbd97698953291732debf9bc7 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/13b71a85-744b-48ba-aecc-2aee2115f8b4_isrForCrosstalkSources_1029418_40.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/13b71a85-744b-48ba-aecc-2aee2115f8b4_isrForCrosstalkSources_1029418_40.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash 924bba4c7c76f4bf0759b40f8edaa93a parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/44478e51-af13-41e4-8e08-142a6b455f5b_isrForCrosstalkSources_1029424_49.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/44478e51-af13-41e4-8e08-142a6b455f5b_isrForCrosstalkSources_1029424_49.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash d67883443f88f38d420df573c3d8c207 parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/00cc46bd-9d37-4e72-8414-ff058746c301_isrForCrosstalkSources_1029425_40.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/00cc46bd-9d37-4e72-8414-ff058746c301_isrForCrosstalkSources_1029425_40.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 64d80a0287cb36210cc48e882cd2a742 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/545d3c60-4bb4-4430-82a3-4937d36b9cfd_isrForCrosstalkSources_1029426_10.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/545d3c60-4bb4-4430-82a3-4937d36b9cfd_isrForCrosstalkSources_1029426_10.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 367a01c6958ae2ba041fe2ed6ac0e413 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/6a83c6cf-f001-4763-808d-3fb7598ddb67_isrForCrosstalkSources_1029424_53.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/6a83c6cf-f001-4763-808d-3fb7598ddb67_isrForCrosstalkSources_1029424_53.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash dc2f6e13c5ebf01127040b22c628a27f parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c20804a6-b261-48fa-bc7b-3a52986e3ea7_isrForCrosstalkSources_1029419_33.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c20804a6-b261-48fa-bc7b-3a52986e3ea7_isrForCrosstalkSources_1029419_33.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 04ba6c48672f128b66696defbd6d583a parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ce865e11-f8cf-46ee-a259-c66d75e67f09_isrForCrosstalkSources_1029425_2.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ce865e11-f8cf-46ee-a259-c66d75e67f09_isrForCrosstalkSources_1029425_2.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 1bf278a172af437b0850c2fc1cf39aef parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8264ff65-7424-407b-9602-e45ca63d0ab6_isrForCrosstalkSources_1029420_62.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8264ff65-7424-407b-9602-e45ca63d0ab6_isrForCrosstalkSources_1029420_62.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash fbe1c78c5cc7986c31bbd2be4e29ceca parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbd9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/a95c6048-2305-469c-8eb9-4e6d79b0d9a7_isrForCrosstalkSources_1029419_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/a95c6048-2305-469c-8eb9-4e6d79b0d9a7_isrForCrosstalkSources_1029419_39.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 0a318e930ce32243d893dd77b254abec parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2c26aca3-21e1-4200-b5e2-9979a1895ea8_isrForCrosstalkSources_1029418_45.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2c26aca3-21e1-4200-b5e2-9979a1895ea8_isrForCrosstalkSources_1029418_45.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 980edb1e95ed5e02b210b7158d4fe528 parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3e34d722-dbf1-4ee6-b707-2c1c9a2dff76_isrForCrosstalkSources_1029418_30.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3e34d722-dbf1-4ee6-b707-2c1c9a2dff76_isrForCrosstalkSources_1029418_30.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 1276cc5750770382bed8076b6b314860 parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e20fa402-d86b-4d8d-bbd8-d5989c76d217_isrForCrosstalkSources_1029418_44.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e20fa402-d86b-4d8d-bbd8-d5989c76d217_isrForCrosstalkSources_1029418_44.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash d3897daa72fba427a06d7141b9df5528 parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3cde1874-0bad-4a21-9196-97c1ab6285a3_isrForCrosstalkSources_1029424_17.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3cde1874-0bad-4a21-9196-97c1ab6285a3_isrForCrosstalkSources_1029424_17.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 743ba5fe2770225675c38d4be1c66aa0 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e395ab4f-4e87-4528-ba9b-373638569d44_isrForCrosstalkSources_1029426_22.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e395ab4f-4e87-4528-ba9b-373638569d44_isrForCrosstalkSources_1029426_22.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 328734f6c9a0a6c3153ac2710d925c1b parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ea705ba9-90fa-4639-9dd0-5fd2473d84c4_isrForCrosstalkSources_1029426_14.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ea705ba9-90fa-4639-9dd0-5fd2473d84c4_isrForCrosstalkSources_1029426_14.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash b57a6d0de07a9c24409b5c41de7f1e80 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/145449f9-dd98-4590-81b5-0140b82b9a54_isrForCrosstalkSources_1029426_37.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/145449f9-dd98-4590-81b5-0140b82b9a54_isrForCrosstalkSources_1029426_37.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash a3721db98f3c3b71b2d3ceb6060dfe5c parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/305ebec3-2f6c-441e-85f8-9617f71b6dde_isrForCrosstalkSources_1029419_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/305ebec3-2f6c-441e-85f8-9617f71b6dde_isrForCrosstalkSources_1029419_61.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 46d9f548fa08991f94ae7ca725b37213 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8d9cfd13-5e95-41d9-8d9a-2677657577f0_isrForCrosstalkSources_1029420_28.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8d9cfd13-5e95-41d9-8d9a-2677657577f0_isrForCrosstalkSources_1029420_28.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash f8d014ea7e81e3bc564748420459d72a parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60c78a95-4609-49e7-8255-01e4cbf03f9d_isrForCrosstalkSources_1029420_45.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60c78a95-4609-49e7-8255-01e4cbf03f9d_isrForCrosstalkSources_1029420_45.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 9538118bc6394b6e72c7be5caabdfb04 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fb82a37d-b5e8-40d3-b0be-e71d2a8b6816_isrForCrosstalkSources_1029418_8.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fb82a37d-b5e8-40d3-b0be-e71d2a8b6816_isrForCrosstalkSources_1029418_8.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash ba277b4976b9be433764e55d2208ba9e parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/97a91374-548c-4e54-ae1f-0c888e1f642a_isrForCrosstalkSources_1029419_1.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/97a91374-548c-4e54-ae1f-0c888e1f642a_isrForCrosstalkSources_1029419_1.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 82e532d488cff2b855afc54f6f63bfa1 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/879b3c1d-7171-429f-8ce6-acc57a48f9de_isrForCrosstalkSources_1029420_32.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/879b3c1d-7171-429f-8ce6-acc57a48f9de_isrForCrosstalkSources_1029420_32.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 598f838efac11217641679bdbab0ac35 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/fbc64140-7af9-428a-8354-07c74e64e269_isrForCrosstalkSources_1029419_34.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/fbc64140-7af9-428a-8354-07c74e64e269_isrForCrosstalkSources_1029419_34.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 616edaa5b69580a83edaaa57c0cb2fb7 parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f5a77c63-c803-4b7e-bc91-edda82a11acb_isrForCrosstalkSources_1029425_41.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f5a77c63-c803-4b7e-bc91-edda82a11acb_isrForCrosstalkSources_1029425_41.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 28c37dbd2b69e4dba49c2a2340dca0c4 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a5adb240-b651-4622-89ac-d235de5565d7_isrForCrosstalkSources_1029420_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a5adb240-b651-4622-89ac-d235de5565d7_isrForCrosstalkSources_1029420_31.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash fbfa08c0eef03a0f7b7eb0de1bdabeeb parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/69b8ed71-51fa-4062-832d-cfbf5cc623e9_isrForCrosstalkSources_1029419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/69b8ed71-51fa-4062-832d-cfbf5cc623e9_isrForCrosstalkSources_1029419_22.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 0a0300a9daa32de4e43062440e9f2cae parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0bf29add-f9a5-49fd-92e1-f6357fed2733_isrForCrosstalkSources_1029420_21.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0bf29add-f9a5-49fd-92e1-f6357fed2733_isrForCrosstalkSources_1029420_21.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 0c1bc923c871efaf92c82d9c2767661c parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/40ea9bb5-1ea6-4560-85ce-4abacaaf740f_isrForCrosstalkSources_1029426_3.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/40ea9bb5-1ea6-4560-85ce-4abacaaf740f_isrForCrosstalkSources_1029426_3.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 12c8c5c7e14a47c63d210b7a41f80250 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10f925ca-dd82-4426-8fe2-220f8f18d5ef_isrForCrosstalkSources_1029425_18.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10f925ca-dd82-4426-8fe2-220f8f18d5ef_isrForCrosstalkSources_1029425_18.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 7e8ed857dda2b76726ca3c4fefd419d0 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c0f30630-b929-444a-bba8-1c3a833094a7_isrForCrosstalkSources_1029420_9.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c0f30630-b929-444a-bba8-1c3a833094a7_isrForCrosstalkSources_1029420_9.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash cc3c3f75f31c4e7ddc23025fed0d8f00 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e02a003f-5731-43cd-a16e-92d50c3b6091_isrForCrosstalkSources_1029424_8.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e02a003f-5731-43cd-a16e-92d50c3b6091_isrForCrosstalkSources_1029424_8.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash ee7c281ce7789f51845f8c70ba6987eb parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbe840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e104756a-fea3-4501-b7b1-740ad98d56f1_isrForCrosstalkSources_1029419_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e104756a-fea3-4501-b7b1-740ad98d56f1_isrForCrosstalkSources_1029419_62.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash 01bb675c956e71f0326bcb0d83956dd8 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbeac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/593af60d-93b7-4cd7-99ca-e842f874a063_isrForCrosstalkSources_1029426_1.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/593af60d-93b7-4cd7-99ca-e842f874a063_isrForCrosstalkSources_1029426_1.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 069765522a41cfc4fae09e0a032c3e8d parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fe646c71-c40b-4296-9570-e9adb1d4fa51_isrForCrosstalkSources_1029425_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fe646c71-c40b-4296-9570-e9adb1d4fa51_isrForCrosstalkSources_1029425_42.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash bff30824dcb643fc0518502724bc65db parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/3fe939e2-e1bd-4207-b828-f8d735061ef8_isrForCrosstalkSources_1029425_46.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/3fe939e2-e1bd-4207-b828-f8d735061ef8_isrForCrosstalkSources_1029425_46.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 339e5e152a12c184ff9bb841760f8d33 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dd148faa-2988-47e1-a101-5c432cc04803_isrForCrosstalkSources_1029424_28.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dd148faa-2988-47e1-a101-5c432cc04803_isrForCrosstalkSources_1029424_28.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 11100903eebf8f9f2c25862ee5061abb parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/25b5d4b3-bb67-4064-8481-01da1ccbbc28_isrForCrosstalkSources_1029419_2.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/25b5d4b3-bb67-4064-8481-01da1ccbbc28_isrForCrosstalkSources_1029419_2.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 6ec00c04396efb8d9d1b45ba0b1d0aea parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e581a876-1b15-4530-b28b-a17723b6fda1_isrForCrosstalkSources_1029425_56.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e581a876-1b15-4530-b28b-a17723b6fda1_isrForCrosstalkSources_1029425_56.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash b54ad6074d6be06772dcfe9dd2440155 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/4b0bf32a-ed46-40ce-8462-a93cf4bd60dc_isrForCrosstalkSources_1029420_51.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/4b0bf32a-ed46-40ce-8462-a93cf4bd60dc_isrForCrosstalkSources_1029420_51.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash e91e3942e3139ed1c4a68bdca917548a parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/df54cab7-250e-4e37-9c2f-c57146686572_isrForCrosstalkSources_1029420_8.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/df54cab7-250e-4e37-9c2f-c57146686572_isrForCrosstalkSources_1029420_8.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash e5cc9bc81165b48f5aafc3ed0e5e5723 parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/d08b29f5-0d07-4302-b735-8fd0e234ff1f_isrForCrosstalkSources_1029426_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/d08b29f5-0d07-4302-b735-8fd0e234ff1f_isrForCrosstalkSources_1029426_25.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 8bd82d8f9fc5c9def2cfc50b0bea4de5 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/88aac6e1-15eb-4022-b179-b38c66046e33_isrForCrosstalkSources_1029419_59.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/88aac6e1-15eb-4022-b179-b38c66046e33_isrForCrosstalkSources_1029419_59.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash d7ebd79baff9aa34f95d958989766819 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e24f7885-b983-4911-a9b0-9879083a46a8_isrForCrosstalkSources_1029425_33.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e24f7885-b983-4911-a9b0-9879083a46a8_isrForCrosstalkSources_1029425_33.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 18b9465712fd6490b4f41e6a3f3ab5ba parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8b719f93-59d4-40d4-a310-6dd5e17ae8ae_isrForCrosstalkSources_1029419_25.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8b719f93-59d4-40d4-a310-6dd5e17ae8ae_isrForCrosstalkSources_1029419_25.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 4c3f814772448e363d6f485900af0358 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/7794b13e-93dc-43da-ac70-b5f7c2ac2885_isrForCrosstalkSources_1029424_3.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/7794b13e-93dc-43da-ac70-b5f7c2ac2885_isrForCrosstalkSources_1029424_3.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 7ecfc369d6f24071ef350f56c3277786 parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/cbb5148c-819e-4e40-9445-7ef112b17552_isrForCrosstalkSources_1029419_12.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/cbb5148c-819e-4e40-9445-7ef112b17552_isrForCrosstalkSources_1029419_12.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 7df4235947d1b9cc7e0937acdfc4634a parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/b8647834-42fe-4a38-b27f-7f683b72359a_isrForCrosstalkSources_1029420_18.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/b8647834-42fe-4a38-b27f-7f683b72359a_isrForCrosstalkSources_1029420_18.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash e298930e73dde0d68374624b55d58184 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/631e997a-fe90-4d1c-84b3-0304dab7d693_isrForCrosstalkSources_1029419_44.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/631e997a-fe90-4d1c-84b3-0304dab7d693_isrForCrosstalkSources_1029419_44.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 4da7971515c3a2fc8c7acac613a61af0 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/052b4ac9-7e45-45a6-8f89-9527367bca5c_isrForCrosstalkSources_1029426_2.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/052b4ac9-7e45-45a6-8f89-9527367bca5c_isrForCrosstalkSources_1029426_2.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash ab892429e0c7274b139d49fc1db51372 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02ec3afd-5c36-47d7-a3ca-dbd1d6eeb8eb_isrForCrosstalkSources_1029419_24.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02ec3afd-5c36-47d7-a3ca-dbd1d6eeb8eb_isrForCrosstalkSources_1029419_24.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 8917a635c6cad56fe4e0a410000ebf49 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a1b3056e-9497-4715-bf58-d2a4bdd586d3_isrForCrosstalkSources_1029420_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a1b3056e-9497-4715-bf58-d2a4bdd586d3_isrForCrosstalkSources_1029420_33.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash 3d35b110ba34738cb18f2b26ebf73753 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0963b21a-2659-4690-87e5-3eb5d8494b25_isrForCrosstalkSources_1029419_23.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0963b21a-2659-4690-87e5-3eb5d8494b25_isrForCrosstalkSources_1029419_23.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash 8ebd63eb6bb9b4b3e874769e36f4e770 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cde14b67-f0e2-4221-ac9f-c5eafd73466f_isrForCrosstalkSources_1029424_33.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cde14b67-f0e2-4221-ac9f-c5eafd73466f_isrForCrosstalkSources_1029424_33.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash 05cdbf722dedf53b512c2327827aaad5 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-16 11:08:31 proc_lsst.multi:170 [INFO] [multi] job_id 17113719 proc_lsst.multi INFO: [multi] job_id 17113719 2024-03-16 11:08:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:08:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:31 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113719 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113719 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17113719 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-16 11:08:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-16 11:08:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:08:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/dc007b7d-1344-459e-8ed1-0672aec17287_isrForCrosstalkSources_1029418_12.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/dc007b7d-1344-459e-8ed1-0672aec17287_isrForCrosstalkSources_1029418_12.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e250d0aedb9d58d779cbd883ba39152b parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/36b85304-6a90-4315-a211-9cc66288aeff_isrForCrosstalkSources_1029418_47.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/36b85304-6a90-4315-a211-9cc66288aeff_isrForCrosstalkSources_1029418_47.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 618c64869606a42b907a879de4364599 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbc0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/239ff6a7-70f3-403a-99bb-b454c0f9f5dd_isrForCrosstalkSources_1029420_16.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/239ff6a7-70f3-403a-99bb-b454c0f9f5dd_isrForCrosstalkSources_1029420_16.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash e63b306d7d3ab1ae3b65eb7bd6431a49 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0c405e83-8b33-48a2-9c0e-4944cad5d2ad_isrForCrosstalkSources_1029420_10.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0c405e83-8b33-48a2-9c0e-4944cad5d2ad_isrForCrosstalkSources_1029420_10.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 887f1d88ee9ff2e9d354b0150ec92619 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0a6d1ada-2153-47d4-a8b5-330f6f782920_isrForCrosstalkSources_1029420_60.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0a6d1ada-2153-47d4-a8b5-330f6f782920_isrForCrosstalkSources_1029420_60.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 2afbc13494ae7b0d966b3c4093cc05e4 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2696d6d4-8ae8-47bb-aac8-34ef0d2bb1e3_isrForCrosstalkSources_1029426_16.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2696d6d4-8ae8-47bb-aac8-34ef0d2bb1e3_isrForCrosstalkSources_1029426_16.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 2f3a8b259f962ef7f5cbbb69ab12a8fb parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d96bca32-b596-4013-9553-24a4859e96f7_isrForCrosstalkSources_1029424_39.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d96bca32-b596-4013-9553-24a4859e96f7_isrForCrosstalkSources_1029424_39.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 42a7a45bd19882252ae0b4dec7f4de26 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/77bd3f6a-59e1-4a3a-92ad-45a2484f7dd9_isrForCrosstalkSources_1029418_28.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/77bd3f6a-59e1-4a3a-92ad-45a2484f7dd9_isrForCrosstalkSources_1029418_28.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash c72ae38f52fc7fada2131ad37c3dbc07 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc93b3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35c26f58-8c15-4769-8a86-3fd14129a481_isrForCrosstalkSources_1029425_47.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35c26f58-8c15-4769-8a86-3fd14129a481_isrForCrosstalkSources_1029425_47.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 307eabfbfd9b722d4415f101a5916f4b parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3dbec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/78a56626-9415-43d0-8799-4c43656c513b_isrForCrosstalkSources_1029418_7.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/78a56626-9415-43d0-8799-4c43656c513b_isrForCrosstalkSources_1029418_7.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash 4eeacd49684d5571f801ec140ab67ffe parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c57420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f0b51f00-e6a9-4348-9ae1-05f6baacdd60_isrForCrosstalkSources_1029418_19.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f0b51f00-e6a9-4348-9ae1-05f6baacdd60_isrForCrosstalkSources_1029418_19.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 7869fe9d2e86e509d4c20f3a69f042a6 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c57240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f9e5a183-b9b9-4536-8d80-137325078df0_isrForCrosstalkSources_1029426_31.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f9e5a183-b9b9-4536-8d80-137325078df0_isrForCrosstalkSources_1029426_31.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 40253b1ee4410cee828cc17f40723cf0 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c540e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/93211b46-9f28-4919-856f-ea8a5919bef6_isrForCrosstalkSources_1029425_23.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/93211b46-9f28-4919-856f-ea8a5919bef6_isrForCrosstalkSources_1029425_23.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 6e023e7b9a8decaeecdb2bec881fb9f4 parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c571a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/eb68501b-9e15-4607-896d-a141ff32bf76_isrForCrosstalkSources_1029420_39.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/eb68501b-9e15-4607-896d-a141ff32bf76_isrForCrosstalkSources_1029420_39.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash d8b7b958b88e0177c0c8aaabe919cfcb parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c542c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e9208400-0594-4a31-a38a-0a09de65de27_isrForCrosstalkSources_1029425_48.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e9208400-0594-4a31-a38a-0a09de65de27_isrForCrosstalkSources_1029425_48.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash a45e65c93549e1c506b6b4e49a85e268 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c54360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/5284632f-5782-41c1-833b-ee342052d9d0_isrForCrosstalkSources_1029426_34.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/5284632f-5782-41c1-833b-ee342052d9d0_isrForCrosstalkSources_1029426_34.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash b5d52f3e24ec9536829bd0f28f1b17c4 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c54220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a3f4d307-6573-45d5-8687-e81a20e1a91e_isrForCrosstalkSources_1029418_21.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a3f4d307-6573-45d5-8687-e81a20e1a91e_isrForCrosstalkSources_1029418_21.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 5d7a9b9bd3cc53f3caf90a2de55db8ad parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c54400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/73517395-c784-4063-aaa5-3297bc17d4ba_isrForCrosstalkSources_1029420_49.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/73517395-c784-4063-aaa5-3297bc17d4ba_isrForCrosstalkSources_1029420_49.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 7acea6a9f2919632e3e139e640cb6553 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c544a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/180fe45b-bb35-4d30-9070-7161dedc4e6f_isrForCrosstalkSources_1029419_52.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/180fe45b-bb35-4d30-9070-7161dedc4e6f_isrForCrosstalkSources_1029419_52.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 2b84ec10ad3c73698c2528185b67cbe6 parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x14afc3c54540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fb8468f7-6230-4492-8c37-c04cd2f83329_isrForCrosstalkSources_1029425_20.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fb8468f7-6230-4492-8c37-c04cd2f83329_isrForCrosstalkSources_1029425_20.stderr 2024-03-16 11:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 17113720 proc_lsst.multi INFO: [multi] job_id 17113720 2024-03-16 11:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113720 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113720 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17113720 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-16 11:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 17113721 proc_lsst.multi INFO: [multi] job_id 17113721 2024-03-16 11:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113721 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113721 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17113721 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-16 11:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:32 proc_lsst.multi:170 [INFO] [multi] job_id 17113722 proc_lsst.multi INFO: [multi] job_id 17113722 2024-03-16 11:08:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:08:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113722 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113722 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17113722 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-16 11:08:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:08:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 17113723 proc_lsst.multi INFO: [multi] job_id 17113723 2024-03-16 11:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 11:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113723 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113723 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17113723 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-16 11:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-16 11:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 11:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:33 proc_lsst.multi:170 [INFO] [multi] job_id 17113724 proc_lsst.multi INFO: [multi] job_id 17113724 2024-03-16 11:08:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 11:08:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:33 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113724 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113724 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17113724 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-16 11:08:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-16 11:08:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 11:08:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:34 proc_lsst.multi:170 [INFO] [multi] job_id 17113725 proc_lsst.multi INFO: [multi] job_id 17113725 2024-03-16 11:08:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 11:08:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113725 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113725 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17113725 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-16 11:08:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-16 11:08:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 11:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:34 proc_lsst.multi:170 [INFO] [multi] job_id 17113726 proc_lsst.multi INFO: [multi] job_id 17113726 2024-03-16 11:08:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 11:08:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:34 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113726 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113726 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17113726 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-16 11:08:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-16 11:08:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 11:08:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:35 proc_lsst.multi:170 [INFO] [multi] job_id 17113727 proc_lsst.multi INFO: [multi] job_id 17113727 2024-03-16 11:08:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 11:08:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:35 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113727 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113727 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17113727 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-16 11:08:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-16 11:08:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 11:08:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 17113728 proc_lsst.multi INFO: [multi] job_id 17113728 2024-03-16 11:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 11:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113728 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113728 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17113728 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-16 11:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-16 11:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 11:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:36 proc_lsst.multi:170 [INFO] [multi] job_id 17113729 proc_lsst.multi INFO: [multi] job_id 17113729 2024-03-16 11:08:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 11:08:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:36 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113729 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113729 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17113729 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-16 11:08:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-16 11:08:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 11:08:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:37 proc_lsst.multi:170 [INFO] [multi] job_id 17113730 proc_lsst.multi INFO: [multi] job_id 17113730 2024-03-16 11:08:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 11:08:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:37 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113730 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113730 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17113730 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-16 11:08:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-16 11:08:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 11:08:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:38 proc_lsst.multi:170 [INFO] [multi] job_id 17113731 proc_lsst.multi INFO: [multi] job_id 17113731 2024-03-16 11:08:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 11:08:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:38 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113731 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113731 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17113731 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-16 11:08:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-16 11:08:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 11:08:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:39 proc_lsst.multi:170 [INFO] [multi] job_id 17113736 proc_lsst.multi INFO: [multi] job_id 17113736 2024-03-16 11:08:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 11:08:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:39 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113736 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113736 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17113736 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-16 11:08:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-16 11:08:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 11:08:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:40 proc_lsst.multi:170 [INFO] [multi] job_id 17113737 proc_lsst.multi INFO: [multi] job_id 17113737 2024-03-16 11:08:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 11:08:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:40 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113737 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113737 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17113737 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-16 11:08:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-16 11:08:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 11:08:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:41 proc_lsst.multi:170 [INFO] [multi] job_id 17113738 proc_lsst.multi INFO: [multi] job_id 17113738 2024-03-16 11:08:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 11:08:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113738 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113738 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17113738 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-16 11:08:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-16 11:08:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 11:08:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:42 proc_lsst.multi:170 [INFO] [multi] job_id 17113739 proc_lsst.multi INFO: [multi] job_id 17113739 2024-03-16 11:08:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 11:08:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113739 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113739 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17113739 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-16 11:08:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-16 11:08:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 11:08:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed92698accb4849085ee0a805e933095 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1f49f0ff-cec1-487d-926d-2f9cea569413_isrForCrosstalkSources_1029418_48.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1f49f0ff-cec1-487d-926d-2f9cea569413_isrForCrosstalkSources_1029418_48.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648ed0970cc65a5a9167e0943a927bce with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bbfe794c-baea-46bd-b82f-2cc1ac3ca9ee_isrForCrosstalkSources_1029418_62.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bbfe794c-baea-46bd-b82f-2cc1ac3ca9ee_isrForCrosstalkSources_1029418_62.stderr 2024-03-16 11:08:43 proc_lsst.multi:170 [INFO] [multi] job_id 17113740 proc_lsst.multi INFO: [multi] job_id 17113740 2024-03-16 11:08:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 11:08:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113740 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113740 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17113740 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-16 11:08:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-16 11:08:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 11:08:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:44 proc_lsst.multi:170 [INFO] [multi] job_id 17113741 proc_lsst.multi INFO: [multi] job_id 17113741 2024-03-16 11:08:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 11:08:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113741 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113741 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17113741 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-16 11:08:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-16 11:08:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 11:08:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 11:08:47 proc_lsst.multi:170 [INFO] [multi] job_id 17113742 proc_lsst.multi INFO: [multi] job_id 17113742 2024-03-16 11:08:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 11:08:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113742 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113742 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17113742 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-16 11:08:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-16 11:08:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 11:08:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff76666de52b286c1d708d98fcd33d0b with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/796c6a06-9931-4c1e-b803-b13366bb16fb_isrForCrosstalkSources_1029418_16.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/796c6a06-9931-4c1e-b803-b13366bb16fb_isrForCrosstalkSources_1029418_16.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc32b568dcde15a82128be69e064dff0 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/4d04e5d0-6b86-4ada-b596-fcb517347123_isrForCrosstalkSources_1029419_43.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/4d04e5d0-6b86-4ada-b596-fcb517347123_isrForCrosstalkSources_1029419_43.stderr 2024-03-16 11:08:48 proc_lsst.multi:170 [INFO] [multi] job_id 17113743 proc_lsst.multi INFO: [multi] job_id 17113743 2024-03-16 11:08:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 11:08:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113743 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113743 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17113743 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-16 11:08:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-16 11:08:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 11:08:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46d90550892f6e87144d5b7230b7ef9 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/94e1dd67-ea60-4b6e-88bc-16a82879f481_isrForCrosstalkSources_1029418_13.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/94e1dd67-ea60-4b6e-88bc-16a82879f481_isrForCrosstalkSources_1029418_13.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f877a99a07c611c21b6070cc651b4b9 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/bdf897ad-ec5c-43e9-b461-41b821d0e0b1_isrForCrosstalkSources_1029419_57.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/bdf897ad-ec5c-43e9-b461-41b821d0e0b1_isrForCrosstalkSources_1029419_57.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 216d3483f124428b327c9027569b5115 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7e19de78-d857-404d-8524-db150f96bda7_isrForCrosstalkSources_1029419_29.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7e19de78-d857-404d-8524-db150f96bda7_isrForCrosstalkSources_1029419_29.stderr 2024-03-16 11:08:50 proc_lsst.multi:170 [INFO] [multi] job_id 17113745 proc_lsst.multi INFO: [multi] job_id 17113745 2024-03-16 11:08:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 11:08:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113745 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113745 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17113745 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-16 11:08:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-16 11:08:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 11:08:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e213f62b473a9cf3f707d27e7ecc1fa with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03886027-e768-4e1f-854c-fd102e81c73c_isrForCrosstalkSources_1029420_26.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03886027-e768-4e1f-854c-fd102e81c73c_isrForCrosstalkSources_1029420_26.stderr 2024-03-16 11:08:51 proc_lsst.multi:170 [INFO] [multi] job_id 17113746 proc_lsst.multi INFO: [multi] job_id 17113746 2024-03-16 11:08:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 11:08:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113746 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113746 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17113746 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-16 11:08:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-16 11:08:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 11:08:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e3e787c339b52bd4da71888cc7e47e4 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8fbdfedb-a9d8-4840-b05d-cb1fb8b91e22_isrForCrosstalkSources_1029419_5.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8fbdfedb-a9d8-4840-b05d-cb1fb8b91e22_isrForCrosstalkSources_1029419_5.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bea79565f7dcf03ae8005dfadb90f89 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/db807573-57af-4e38-85fb-fa3ce0e2e64a_isrForCrosstalkSources_1029419_20.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/db807573-57af-4e38-85fb-fa3ce0e2e64a_isrForCrosstalkSources_1029419_20.stderr 2024-03-16 11:08:53 proc_lsst.multi:170 [INFO] [multi] job_id 17113747 proc_lsst.multi INFO: [multi] job_id 17113747 2024-03-16 11:08:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 11:08:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113747 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113747 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17113747 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-16 11:08:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-16 11:08:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 11:08:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d29a67e1e53543b6367ba54b8984b2e5 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ad347104-d1cf-4cf1-ae1f-8f0aad801889_isrForCrosstalkSources_1029424_21.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ad347104-d1cf-4cf1-ae1f-8f0aad801889_isrForCrosstalkSources_1029424_21.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4152f319c46e7cab97b6ea9c00f4868f with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/eadf54b9-1b20-4e72-9c6e-7052cf305c76_isrForCrosstalkSources_1029424_42.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/eadf54b9-1b20-4e72-9c6e-7052cf305c76_isrForCrosstalkSources_1029424_42.stderr 2024-03-16 11:08:54 proc_lsst.multi:170 [INFO] [multi] job_id 17113748 proc_lsst.multi INFO: [multi] job_id 17113748 2024-03-16 11:08:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 11:08:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113748 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113748 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17113748 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-16 11:08:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-16 11:08:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 11:08:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ce59facdf5a4adb624a09054362186 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/56cfcab2-5675-421e-8cb5-d51a397ac745_isrForCrosstalkSources_1029426_54.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/56cfcab2-5675-421e-8cb5-d51a397ac745_isrForCrosstalkSources_1029426_54.stderr 2024-03-16 11:08:56 proc_lsst.multi:170 [INFO] [multi] job_id 17113749 proc_lsst.multi INFO: [multi] job_id 17113749 2024-03-16 11:08:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 11:08:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113749 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113749 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17113749 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-16 11:08:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-16 11:08:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 11:08:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d061fe522ed1ce5c92d3b0035832977e with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a0da7dde-da8c-4980-b9c3-8e7567fb5c22_isrForCrosstalkSources_1029424_2.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a0da7dde-da8c-4980-b9c3-8e7567fb5c22_isrForCrosstalkSources_1029424_2.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 670fdd808dba65bae9c922494c15af8f with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4a6398af-cbe6-40d6-bd62-eb16e25d8189_isrForCrosstalkSources_1029425_30.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4a6398af-cbe6-40d6-bd62-eb16e25d8189_isrForCrosstalkSources_1029425_30.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5c40dc2097f2b2b6523a1d64fb56782 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f7a02712-f5e2-4b3e-8a4b-a150a99eee1f_isrForCrosstalkSources_1029420_34.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f7a02712-f5e2-4b3e-8a4b-a150a99eee1f_isrForCrosstalkSources_1029420_34.stderr 2024-03-16 11:08:57 proc_lsst.multi:170 [INFO] [multi] job_id 17113750 proc_lsst.multi INFO: [multi] job_id 17113750 2024-03-16 11:08:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 11:08:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113750 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113750 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17113750 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-16 11:08:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-16 11:08:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 11:08:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d5b528555d283e711da7ecae9c62961 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd2f294-993c-4daa-b2d7-37322c4b2ad5_isrForCrosstalkSources_1029420_13.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd2f294-993c-4daa-b2d7-37322c4b2ad5_isrForCrosstalkSources_1029420_13.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a26a128a09f6b443b5b81f06a860bf0 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87413a7b-a034-4b1b-945d-ce0f07616f7a_isrForCrosstalkSources_1029424_31.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87413a7b-a034-4b1b-945d-ce0f07616f7a_isrForCrosstalkSources_1029424_31.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25f8dc0cd9f76830e025a415436ecdcc with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b5feb600-7c6e-4a04-b7bf-c3dfd1a1a802_isrForCrosstalkSources_1029418_36.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b5feb600-7c6e-4a04-b7bf-c3dfd1a1a802_isrForCrosstalkSources_1029418_36.stderr 2024-03-16 11:08:59 proc_lsst.multi:170 [INFO] [multi] job_id 17113751 proc_lsst.multi INFO: [multi] job_id 17113751 2024-03-16 11:08:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 11:08:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113751 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113751 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17113751 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-16 11:08:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-16 11:08:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:08:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:08:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 11:08:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:08:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4ae10c24e119adb52cf0cd5dcca586c with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/4e470592-8e71-45db-8b2d-d431acd1d9a3_isrForCrosstalkSources_1029424_45.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/4e470592-8e71-45db-8b2d-d431acd1d9a3_isrForCrosstalkSources_1029424_45.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 818a3e91b7f4f420a887195bff570d70 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/7f2d354a-ba99-4537-a8ca-6b28a43466a2_isrForCrosstalkSources_1029420_17.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/7f2d354a-ba99-4537-a8ca-6b28a43466a2_isrForCrosstalkSources_1029420_17.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4e96d11b7185605a6499e8466a06585 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/cf44b1b2-3c8d-470f-bfbe-0c41ed783fc9_isrForCrosstalkSources_1029426_21.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/cf44b1b2-3c8d-470f-bfbe-0c41ed783fc9_isrForCrosstalkSources_1029426_21.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ef065e34d60dcc3247679cc7ac9b27a with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/633b1afe-d5a6-4518-b445-8de384b96fa7_isrForCrosstalkSources_1029426_18.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/633b1afe-d5a6-4518-b445-8de384b96fa7_isrForCrosstalkSources_1029426_18.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12eab8f74e970734440cd89b76c39550 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/811225b2-f7a1-4993-8b4a-68a0d059eba9_isrForCrosstalkSources_1029420_55.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/811225b2-f7a1-4993-8b4a-68a0d059eba9_isrForCrosstalkSources_1029420_55.stderr 2024-03-16 11:09:01 proc_lsst.multi:170 [INFO] [multi] job_id 17113752 proc_lsst.multi INFO: [multi] job_id 17113752 2024-03-16 11:09:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17113752 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17113752 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17113752 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-16 11:09:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-16 11:09:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:01 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 11:09:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfc2f580254e40eaaf1ac1103ec89016 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/20a9f84a-d26b-4fb2-8034-dba1c57e6300_isrForCrosstalkSources_1029418_61.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/20a9f84a-d26b-4fb2-8034-dba1c57e6300_isrForCrosstalkSources_1029418_61.stderr 2024-03-16 11:09:03 proc_lsst.multi:170 [INFO] [multi] job_id 17113754 proc_lsst.multi INFO: [multi] job_id 17113754 2024-03-16 11:09:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:03 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17113754 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17113754 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17113754 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-16 11:09:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-16 11:09:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 769e99723415a9530a68dc20aa67b779 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/631c919a-a540-4574-bbbe-7e332dc484e4_isrForCrosstalkSources_1029424_7.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/631c919a-a540-4574-bbbe-7e332dc484e4_isrForCrosstalkSources_1029424_7.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7402aeb76ee949edc742101e5d0efb6 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/847b842f-23ac-4a75-b8a9-cd5257781cd6_isrForCrosstalkSources_1029418_2.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/847b842f-23ac-4a75-b8a9-cd5257781cd6_isrForCrosstalkSources_1029418_2.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c3276ddb745ff3311bf3444e9f6b2f with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93b1afab-1516-4106-8846-1c0e8dfdd83a_isrForCrosstalkSources_1029419_35.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93b1afab-1516-4106-8846-1c0e8dfdd83a_isrForCrosstalkSources_1029419_35.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbbfbc4abeca422bd077d9639adf2417 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0a0d00b4-a61f-4940-8f63-34ba7e7d0697_isrForCrosstalkSources_1029424_36.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0a0d00b4-a61f-4940-8f63-34ba7e7d0697_isrForCrosstalkSources_1029424_36.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a4e84f73e7279139180a094befa695d with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7d11ebcd-b010-48da-8285-fb5db08f19ae_isrForCrosstalkSources_1029426_30.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7d11ebcd-b010-48da-8285-fb5db08f19ae_isrForCrosstalkSources_1029426_30.stderr 2024-03-16 11:09:05 proc_lsst.multi:170 [INFO] [multi] job_id 17113756 proc_lsst.multi INFO: [multi] job_id 17113756 2024-03-16 11:09:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:09:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:05 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17113756 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17113756 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17113756 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-16 11:09:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-16 11:09:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:09:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39df00fd15ea7cb41d2fea30c903f51b with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a5802c0a-f0d7-48cc-b741-68985452ba18_isrForCrosstalkSources_1029424_35.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/a5802c0a-f0d7-48cc-b741-68985452ba18_isrForCrosstalkSources_1029424_35.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70bbd1bb91e8657a9ae5fa3233ac3312 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f2999621-497e-4688-af59-291c2ed5bc2a_isrForCrosstalkSources_1029418_22.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f2999621-497e-4688-af59-291c2ed5bc2a_isrForCrosstalkSources_1029418_22.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10e061c8eb271d704072c0e0255b4f24 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/3f20170a-96f2-49e2-8487-4763f16f4982_isrForCrosstalkSources_1029419_3.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/3f20170a-96f2-49e2-8487-4763f16f4982_isrForCrosstalkSources_1029419_3.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29391f2ddd44ce8d1841577c871c76ca with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9104638f-5b4d-40a3-8206-b06875361b6f_isrForCrosstalkSources_1029426_49.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9104638f-5b4d-40a3-8206-b06875361b6f_isrForCrosstalkSources_1029426_49.stderr 2024-03-16 11:09:07 proc_lsst.multi:170 [INFO] [multi] job_id 17113757 proc_lsst.multi INFO: [multi] job_id 17113757 2024-03-16 11:09:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:09:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:07 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17113757 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17113757 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17113757 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-16 11:09:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-16 11:09:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:09:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db57fb42b41c4d9143451c90a5b8dff3 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4421e02b-ed01-47e2-90f8-ae606251d19a_isrForCrosstalkSources_1029426_60.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4421e02b-ed01-47e2-90f8-ae606251d19a_isrForCrosstalkSources_1029426_60.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 811153df4f90b24f984e67ef23b61502 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2645ecfb-8c6b-4678-a1bf-d7f43334dcd7_isrForCrosstalkSources_1029418_46.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2645ecfb-8c6b-4678-a1bf-d7f43334dcd7_isrForCrosstalkSources_1029418_46.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e999e871774ae3f3635db0055e5b9461 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3931456c-f10b-439a-b5b7-ea3659f613c3_isrForCrosstalkSources_1029424_58.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3931456c-f10b-439a-b5b7-ea3659f613c3_isrForCrosstalkSources_1029424_58.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b646b7b3133c79e7d77ac53054e906f with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d97c063c-0cac-4cb4-915f-2422036b5ae9_isrForCrosstalkSources_1029419_38.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d97c063c-0cac-4cb4-915f-2422036b5ae9_isrForCrosstalkSources_1029419_38.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35499d557048a9336b242ebe4e6609d5 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/216c3002-c5bd-493d-addf-15f20850f6ea_isrForCrosstalkSources_1029419_46.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/216c3002-c5bd-493d-addf-15f20850f6ea_isrForCrosstalkSources_1029419_46.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4305dfa3c839a97de8917c279ac16249 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/7e184b7e-5750-45cb-88ae-2aa0cbff8248_isrForCrosstalkSources_1029418_18.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/7e184b7e-5750-45cb-88ae-2aa0cbff8248_isrForCrosstalkSources_1029418_18.stderr 2024-03-16 11:09:10 proc_lsst.multi:170 [INFO] [multi] job_id 17113763 proc_lsst.multi INFO: [multi] job_id 17113763 2024-03-16 11:09:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:09:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:10 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17113763 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17113763 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17113763 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-16 11:09:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-16 11:09:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:09:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a635b1987e831c7a1464dca7f7d7c432 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c4382375-d083-4d70-90c4-ef7576bffd2d_isrForCrosstalkSources_1029426_47.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c4382375-d083-4d70-90c4-ef7576bffd2d_isrForCrosstalkSources_1029426_47.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a518db878a9ad022b353531cfe5c239 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f51c09fa-ce0d-423b-80ce-1c594c52cedf_isrForCrosstalkSources_1029426_20.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f51c09fa-ce0d-423b-80ce-1c594c52cedf_isrForCrosstalkSources_1029426_20.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 602f038508a501820090738d7756a4e9 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e8fddf3b-5e35-4a72-b99a-b791d9824d27_isrForCrosstalkSources_1029426_11.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e8fddf3b-5e35-4a72-b99a-b791d9824d27_isrForCrosstalkSources_1029426_11.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db681d53f5e3a88891f6394b92b57eda with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/05b620d7-f6bd-4c1f-9375-cc3cf4509802_isrForCrosstalkSources_1029418_23.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/05b620d7-f6bd-4c1f-9375-cc3cf4509802_isrForCrosstalkSources_1029418_23.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8182c4d8d5b651b934b64b612185565 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/02ba2d26-ffbd-44ad-8aee-b8f73bda1398_isrForCrosstalkSources_1029426_4.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/02ba2d26-ffbd-44ad-8aee-b8f73bda1398_isrForCrosstalkSources_1029426_4.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cca929f640725bb8036597c7a63dcf4 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c6f4633c-605a-4c6a-9677-2b08d0177d70_isrForCrosstalkSources_1029418_50.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c6f4633c-605a-4c6a-9677-2b08d0177d70_isrForCrosstalkSources_1029418_50.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94af40601435f62be768d31a38707eb7 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b187ed86-4268-4358-b96d-1a5f3466e61b_isrForCrosstalkSources_1029424_13.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b187ed86-4268-4358-b96d-1a5f3466e61b_isrForCrosstalkSources_1029424_13.stderr 2024-03-16 11:09:14 proc_lsst.multi:170 [INFO] [multi] job_id 17113764 proc_lsst.multi INFO: [multi] job_id 17113764 2024-03-16 11:09:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:14 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17113764 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17113764 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17113764 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-16 11:09:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-16 11:09:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 11:09:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f33c6705493a59bbeda4caef2ab08560 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd130870-16e8-4622-b63e-70579b3837bf_isrForCrosstalkSources_1029425_26.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd130870-16e8-4622-b63e-70579b3837bf_isrForCrosstalkSources_1029425_26.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b265e3fc8bc178fedacdd8ae0b9f54f7 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7592382b-2dbd-4770-8382-363c96477d74_isrForCrosstalkSources_1029425_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7592382b-2dbd-4770-8382-363c96477d74_isrForCrosstalkSources_1029425_34.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a05f193a2abdb4cd9b911ed11815e8a with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/808af997-6e10-4a5c-a704-a4a3235ad4f4_isrForCrosstalkSources_1029426_17.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/808af997-6e10-4a5c-a704-a4a3235ad4f4_isrForCrosstalkSources_1029426_17.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 918211de4671497dc2a351b9ef34b393 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/5799789a-3975-4d62-8830-f67f1bdedde8_isrForCrosstalkSources_1029425_27.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/5799789a-3975-4d62-8830-f67f1bdedde8_isrForCrosstalkSources_1029425_27.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4425525d1fd3f41e9cda61af9ea1d48 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/72ab27b8-1407-458f-8627-83c8b3c0ec75_isrForCrosstalkSources_1029420_43.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/72ab27b8-1407-458f-8627-83c8b3c0ec75_isrForCrosstalkSources_1029420_43.stderr 2024-03-16 11:09:16 proc_lsst.multi:170 [INFO] [multi] job_id 17113766 proc_lsst.multi INFO: [multi] job_id 17113766 2024-03-16 11:09:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113766 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113766 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17113766 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-16 11:09:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-16 11:09:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e59582a4bd59a4895fddddb9a516810 with result from task 53 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8f993352-e07d-40ea-9097-8a90d32819a7_isrForCrosstalkSources_1029420_53.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8f993352-e07d-40ea-9097-8a90d32819a7_isrForCrosstalkSources_1029420_53.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb7e1cab447bbfe9bc79723bf3a23772 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c680fe76-ebdd-449d-9fa0-4c20c857fbfc_isrForCrosstalkSources_1029424_18.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c680fe76-ebdd-449d-9fa0-4c20c857fbfc_isrForCrosstalkSources_1029424_18.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b16ead9c54aaa21856bbca4e518c94f with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0c5d61e8-1a91-482f-90bf-77299c1aafc9_isrForCrosstalkSources_1029424_32.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0c5d61e8-1a91-482f-90bf-77299c1aafc9_isrForCrosstalkSources_1029424_32.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61acb379031e89d72526360b5006b0a4 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/096d1c6b-fb9a-4e21-bbf6-6116d4c21814_isrForCrosstalkSources_1029419_51.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/096d1c6b-fb9a-4e21-bbf6-6116d4c21814_isrForCrosstalkSources_1029419_51.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 444d6175b3b8f17d43d310dcb470ce66 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c042330c-32b2-4c5e-94a4-21ee97eb161d_isrForCrosstalkSources_1029424_55.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c042330c-32b2-4c5e-94a4-21ee97eb161d_isrForCrosstalkSources_1029424_55.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e6b62455d619e1db3fcf1f4aa85d15 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3f259590-43f2-4904-b482-f66f7b57a11d_isrForCrosstalkSources_1029424_60.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3f259590-43f2-4904-b482-f66f7b57a11d_isrForCrosstalkSources_1029424_60.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99aa2b6def1427f552da6bb72d05db20 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a1e5a3c5-2704-490b-96af-f44ebbc9b8c5_isrForCrosstalkSources_1029418_1.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a1e5a3c5-2704-490b-96af-f44ebbc9b8c5_isrForCrosstalkSources_1029418_1.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb2d57d23a662b09f5d523cc270e9b52 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9cd66295-10e0-4644-aa6a-718e88f7bf08_isrForCrosstalkSources_1029418_5.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9cd66295-10e0-4644-aa6a-718e88f7bf08_isrForCrosstalkSources_1029418_5.stderr 2024-03-16 11:09:19 proc_lsst.multi:170 [INFO] [multi] job_id 17113767 proc_lsst.multi INFO: [multi] job_id 17113767 2024-03-16 11:09:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:09:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113767 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113767 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17113767 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-16 11:09:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-16 11:09:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 11:09:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e6465a90652391b1d4849143f25e1c0 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93c54dff-bc02-4a01-8fc7-7d083a175925_isrForCrosstalkSources_1029419_26.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/93c54dff-bc02-4a01-8fc7-7d083a175925_isrForCrosstalkSources_1029419_26.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06684a30bd419eb007e40cf00f8c66bb with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/402b3493-39a0-4b3f-a9d6-12dbe638b725_isrForCrosstalkSources_1029425_12.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/402b3493-39a0-4b3f-a9d6-12dbe638b725_isrForCrosstalkSources_1029425_12.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b57da4797431a98b241aa945d795c70 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/6c532ab1-7b9a-4bf4-8843-158a9115c989_isrForCrosstalkSources_1029418_54.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/6c532ab1-7b9a-4bf4-8843-158a9115c989_isrForCrosstalkSources_1029418_54.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4c363b3ed17fe4f9bc0fbbb17a2fc80 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/1eccc6b6-31ed-4ffc-8187-b1e816a5d170_isrForCrosstalkSources_1029420_42.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/1eccc6b6-31ed-4ffc-8187-b1e816a5d170_isrForCrosstalkSources_1029420_42.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eadd139dfded25d0c6823e43f10511f8 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/76f7ed00-32ea-4058-84c7-6f2cdc5a3e88_isrForCrosstalkSources_1029418_57.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/76f7ed00-32ea-4058-84c7-6f2cdc5a3e88_isrForCrosstalkSources_1029418_57.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011d8891ed92d487634ed4835406384a with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7b27a528-9110-46b2-a0cb-7e774b86f3b5_isrForCrosstalkSources_1029426_42.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7b27a528-9110-46b2-a0cb-7e774b86f3b5_isrForCrosstalkSources_1029426_42.stderr 2024-03-16 11:09:20 proc_lsst.multi:170 [INFO] [multi] job_id 17113768 proc_lsst.multi INFO: [multi] job_id 17113768 2024-03-16 11:09:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:09:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113768 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113768 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17113768 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-16 11:09:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-16 11:09:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 11:09:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea63710649086af04f658d0d5bf9ae79 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/622a937d-fbec-481e-9f81-92bc9cbb405c_isrForCrosstalkSources_1029425_37.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/622a937d-fbec-481e-9f81-92bc9cbb405c_isrForCrosstalkSources_1029425_37.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcc46cec871e8e355ff2430330abfcd5 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3bf0eaab-b31b-49e4-97f6-ae18c4277246_isrForCrosstalkSources_1029418_15.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3bf0eaab-b31b-49e4-97f6-ae18c4277246_isrForCrosstalkSources_1029418_15.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3095ad49eedfd7ce84e7f824ebaffd8e with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/240de32f-129b-41fb-8090-d584a511e1a0_isrForCrosstalkSources_1029419_13.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/240de32f-129b-41fb-8090-d584a511e1a0_isrForCrosstalkSources_1029419_13.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cae50125a331a14e85e136cb557b70a with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/74cb822c-a544-4912-9416-a617378ab0d9_isrForCrosstalkSources_1029418_11.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/74cb822c-a544-4912-9416-a617378ab0d9_isrForCrosstalkSources_1029418_11.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5e81b05653eb5ec3dfa48122f527804 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7759b3fd-3ff6-4d17-8280-0db966bfb0aa_isrForCrosstalkSources_1029425_43.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7759b3fd-3ff6-4d17-8280-0db966bfb0aa_isrForCrosstalkSources_1029425_43.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c4534b5edda2964d34b357c9d81d9fb with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/410a300b-ff87-4029-92bf-0d9bfb64600f_isrForCrosstalkSources_1029419_32.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/410a300b-ff87-4029-92bf-0d9bfb64600f_isrForCrosstalkSources_1029419_32.stderr 2024-03-16 11:09:22 proc_lsst.multi:170 [INFO] [multi] job_id 17113769 proc_lsst.multi INFO: [multi] job_id 17113769 2024-03-16 11:09:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:09:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:22 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113769 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113769 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17113769 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-16 11:09:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-16 11:09:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:22 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 11:09:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2caeadd4200762a255d6c2bd7dce2393 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/143e882f-792f-481f-9950-4a85fb870888_isrForCrosstalkSources_1029424_15.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/143e882f-792f-481f-9950-4a85fb870888_isrForCrosstalkSources_1029424_15.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43f84b991c665e56ba9f768157c75e2b with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c9c2d679-3780-4039-a462-b356c5092f77_isrForCrosstalkSources_1029420_56.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c9c2d679-3780-4039-a462-b356c5092f77_isrForCrosstalkSources_1029420_56.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 724b3dbe01ef8a9ca391180a701bf0c4 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d47d3f6a-6ae3-4463-b35e-cabe60fdb651_isrForCrosstalkSources_1029424_51.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d47d3f6a-6ae3-4463-b35e-cabe60fdb651_isrForCrosstalkSources_1029424_51.stderr 2024-03-16 11:09:24 proc_lsst.multi:170 [INFO] [multi] job_id 17113770 proc_lsst.multi INFO: [multi] job_id 17113770 2024-03-16 11:09:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113770 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113770 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17113770 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-16 11:09:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-16 11:09:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 11:09:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 11:09:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 11:09:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 11:09:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 11:09:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 11:09:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 11:09:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 11:09:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,169.254.95.120,10.64.65.9,198.48.92.26,n3009,10.64.129.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54680 --result_port=54309 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f9cd725bd2cbdf78a6add41e118a1e3 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/2ae9c992-1f6b-4397-9244-d16541a3c4e9_isrForCrosstalkSources_1029424_24.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/2ae9c992-1f6b-4397-9244-d16541a3c4e9_isrForCrosstalkSources_1029424_24.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 592c68720b31e68338c9f42b31a010c2 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c48d3481-b1ea-4ffe-b972-b4fef90b0991_isrForCrosstalkSources_1029419_40.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c48d3481-b1ea-4ffe-b972-b4fef90b0991_isrForCrosstalkSources_1029419_40.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc519a18a74268085468f31d5780e98e with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f0542578-842d-432a-ab0e-45ec0f860020_isrForCrosstalkSources_1029420_6.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/f0542578-842d-432a-ab0e-45ec0f860020_isrForCrosstalkSources_1029420_6.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a938c3a854fe53f982936b0c8a273c63 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03fbbdaa-4201-4f18-9cfd-b8cfdf9248ed_isrForCrosstalkSources_1029420_35.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/03fbbdaa-4201-4f18-9cfd-b8cfdf9248ed_isrForCrosstalkSources_1029420_35.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47a37c8cbf5076ebbabebdedfdf0a10e with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1c38d82c-cf82-4368-8299-3ec1d1f34e93_isrForCrosstalkSources_1029424_12.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1c38d82c-cf82-4368-8299-3ec1d1f34e93_isrForCrosstalkSources_1029424_12.stderr 2024-03-16 11:09:27 proc_lsst.multi:170 [INFO] [multi] job_id 17113771 proc_lsst.multi INFO: [multi] job_id 17113771 2024-03-16 11:09:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 11:09:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 11:09:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17113771 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17113771 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17113771 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22745230885392 2024-03-16 11:09:27 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local 2024-03-16 11:09:27 proc_lsst.multi:146 [INFO] found job 17113718 in provider astro proc_lsst.multi INFO: found job 17113718 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ae69c9a9683414151912fa1d1e88bc with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1b02517c-94be-46f3-b1da-e0a1dcb1e255_isrForCrosstalkSources_1029424_41.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1b02517c-94be-46f3-b1da-e0a1dcb1e255_isrForCrosstalkSources_1029424_41.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee1a90672e415725c7b43c7f7a51fbd with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/da52466c-fca0-49b9-afc8-b28a823c93f6_isrForCrosstalkSources_1029424_20.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/da52466c-fca0-49b9-afc8-b28a823c93f6_isrForCrosstalkSources_1029424_20.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13d5d54e79ced6ee653f3857d5911930 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3759c4b9-e7f1-496e-a3b0-e37451cd4692_isrForCrosstalkSources_1029418_26.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3759c4b9-e7f1-496e-a3b0-e37451cd4692_isrForCrosstalkSources_1029418_26.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c0d35849743847fecd1194bd2afc72c with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7f00980f-1980-4c1c-b787-96ce599240c5_isrForCrosstalkSources_1029419_18.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/7f00980f-1980-4c1c-b787-96ce599240c5_isrForCrosstalkSources_1029419_18.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58a2564ef784f3d01489b1786bd21cc4 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/61860506-8c30-4163-8545-df9c6cfabed9_isrForCrosstalkSources_1029426_46.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/61860506-8c30-4163-8545-df9c6cfabed9_isrForCrosstalkSources_1029426_46.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d0c0b0c6289a89e8f9d8ad6879c822a with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/f966c98b-a75d-4ed3-ac0d-675b2ea3d2f3_isrForCrosstalkSources_1029419_53.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/f966c98b-a75d-4ed3-ac0d-675b2ea3d2f3_isrForCrosstalkSources_1029419_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:29 proc_lsst.multi:146 [INFO] found job 17113719 in provider astro proc_lsst.multi INFO: found job 17113719 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b7c47f84bfec2b7b9605d7b4681c4f2 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7c9917c8-80b1-46d2-b39f-f88a90fddb63_isrForCrosstalkSources_1029425_1.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7c9917c8-80b1-46d2-b39f-f88a90fddb63_isrForCrosstalkSources_1029425_1.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ba5b418e0fc824f402578406326531d with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6e6f29f8-20fb-4bdd-bd2a-e5dc21f82b12_isrForCrosstalkSources_1029425_50.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6e6f29f8-20fb-4bdd-bd2a-e5dc21f82b12_isrForCrosstalkSources_1029425_50.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc82765efa5fa984172174387aef1f96 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a9fcf7bc-24bd-437e-ad53-a6a337b12e13_isrForCrosstalkSources_1029425_36.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a9fcf7bc-24bd-437e-ad53-a6a337b12e13_isrForCrosstalkSources_1029425_36.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 196e83ede0031f93fe087a58c6f649d7 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/40b1f608-b2fb-4459-9d82-027a8ace2756_isrForCrosstalkSources_1029419_14.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/40b1f608-b2fb-4459-9d82-027a8ace2756_isrForCrosstalkSources_1029419_14.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 701d904d97328be0eff13d40eb893c3b with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c026084c-20dc-45f6-a8f6-22ccd9f0ca38_isrForCrosstalkSources_1029424_38.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c026084c-20dc-45f6-a8f6-22ccd9f0ca38_isrForCrosstalkSources_1029424_38.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d08e6e7d0cfe5b7c499bf8e220875667 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/d16c4f16-6f8e-4c0d-a962-81b069ef4983_isrForCrosstalkSources_1029420_40.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/d16c4f16-6f8e-4c0d-a962-81b069ef4983_isrForCrosstalkSources_1029420_40.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65f418b9234fa34a7d9596407961b091 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/a6297901-7f69-42ba-b03e-10093e585c43_isrForCrosstalkSources_1029426_35.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/a6297901-7f69-42ba-b03e-10093e585c43_isrForCrosstalkSources_1029426_35.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ed33836e8a9938f0836d9ad27afacfc with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/59924cdb-6824-4977-ad4e-95425fe62e60_isrForCrosstalkSources_1029426_15.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/59924cdb-6824-4977-ad4e-95425fe62e60_isrForCrosstalkSources_1029426_15.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 390bbd6383571eba1c0dcf829619d0e9 with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c40042c0-a78b-4bd7-a42f-8aaa42f2c35d_isrForCrosstalkSources_1029424_14.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c40042c0-a78b-4bd7-a42f-8aaa42f2c35d_isrForCrosstalkSources_1029424_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:32 proc_lsst.multi:146 [INFO] found job 17113720 in provider astro proc_lsst.multi INFO: found job 17113720 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3f34589643f6288c25f78ec50223633 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/eb53e863-1cd1-4072-ace7-e20d642a5e7c_isrForCrosstalkSources_1029426_52.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/eb53e863-1cd1-4072-ace7-e20d642a5e7c_isrForCrosstalkSources_1029426_52.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c65dcc886e4b9e2f8b3f6ccacbcc390b with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8acee2d4-a270-4b7c-8523-9ee65ff9a635_isrForCrosstalkSources_1029424_48.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8acee2d4-a270-4b7c-8523-9ee65ff9a635_isrForCrosstalkSources_1029424_48.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9df2653ef892eb43bbb161adde2c721 with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bf8617a5-0575-46af-8f8c-7f1abe0bc4b6_isrForCrosstalkSources_1029424_37.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bf8617a5-0575-46af-8f8c-7f1abe0bc4b6_isrForCrosstalkSources_1029424_37.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a7bc7797dd7c4a94a00d42825d875a8 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ebf4b53e-bd7d-4af2-bc6f-98148f426a9e_isrForCrosstalkSources_1029425_28.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ebf4b53e-bd7d-4af2-bc6f-98148f426a9e_isrForCrosstalkSources_1029425_28.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ce99faa145cbd9975b5f6afee4c6bd4 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9c5ac8d1-d0c0-4ad8-816f-73afc481fb81_isrForCrosstalkSources_1029418_37.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/9c5ac8d1-d0c0-4ad8-816f-73afc481fb81_isrForCrosstalkSources_1029418_37.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67155995f0767d9bc7c7d6e359a5f7a1 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/53ea78e8-42ba-4e56-b5c2-a20365cbf7c4_isrForCrosstalkSources_1029418_43.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/53ea78e8-42ba-4e56-b5c2-a20365cbf7c4_isrForCrosstalkSources_1029418_43.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b61e208ab1398a5c7d9cb60d12cf937a with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/57624f3e-9976-41db-9136-14dfc05a1b83_isrForCrosstalkSources_1029419_6.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/57624f3e-9976-41db-9136-14dfc05a1b83_isrForCrosstalkSources_1029419_6.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a5a6463c111be0acd187096d31449fc with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/71e10703-60a4-4fca-bfda-b011846dcc1f_isrForCrosstalkSources_1029420_23.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/71e10703-60a4-4fca-bfda-b011846dcc1f_isrForCrosstalkSources_1029420_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:34 proc_lsst.multi:146 [INFO] found job 17113721 in provider astro proc_lsst.multi INFO: found job 17113721 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47944b4e7feae9c012e4852be2445f71 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/852ce436-aec8-48b6-9fdb-d92e122ce483_isrForCrosstalkSources_1029419_50.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/852ce436-aec8-48b6-9fdb-d92e122ce483_isrForCrosstalkSources_1029419_50.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 697cd56696ab8717fdf96f2d48a6619e with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0844495a-a046-4c0a-8e16-31ec18d51a29_isrForCrosstalkSources_1029419_47.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0844495a-a046-4c0a-8e16-31ec18d51a29_isrForCrosstalkSources_1029419_47.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c909789fde939502f4b1feb82edd3467 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7dc23b0e-4d11-456e-a62b-725b4f0d76e7_isrForCrosstalkSources_1029426_61.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7dc23b0e-4d11-456e-a62b-725b4f0d76e7_isrForCrosstalkSources_1029426_61.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43c3e4395cde9cbef370516c92736e5c with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c497c9b0-4e95-4eb1-8e9a-4df9f94604d5_isrForCrosstalkSources_1029419_30.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c497c9b0-4e95-4eb1-8e9a-4df9f94604d5_isrForCrosstalkSources_1029419_30.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c281b0fd164298d83082b5658853f9f with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/715c7c68-7228-4c1b-b1cd-940f7fabe9ac_isrForCrosstalkSources_1029420_37.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/715c7c68-7228-4c1b-b1cd-940f7fabe9ac_isrForCrosstalkSources_1029420_37.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 684214d7b8147e22055709a0b7e4fc29 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8b69c480-874d-4a14-a6a0-c4c796dee1db_isrForCrosstalkSources_1029420_57.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8b69c480-874d-4a14-a6a0-c4c796dee1db_isrForCrosstalkSources_1029420_57.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bff606b0f05de8279dafa03194dd74a with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/71ae7561-9de1-4908-ba46-e9701c994a5d_isrForCrosstalkSources_1029418_17.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/71ae7561-9de1-4908-ba46-e9701c994a5d_isrForCrosstalkSources_1029418_17.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7a625ab7161d3ebdf1f15e528b05fc8 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2709f67c-66b3-4312-adf0-bde1057ce0ea_isrForCrosstalkSources_1029425_31.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2709f67c-66b3-4312-adf0-bde1057ce0ea_isrForCrosstalkSources_1029425_31.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06f2996dd27fb4fcf7050b86a8cc09b with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/64441d16-e240-4a0f-9f0a-73184757d16e_isrForCrosstalkSources_1029424_9.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/64441d16-e240-4a0f-9f0a-73184757d16e_isrForCrosstalkSources_1029424_9.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f26fa338fbab78e9fe055d7f8089c5ec with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6eb2923a-425d-4a36-b4e2-8121e105efc5_isrForCrosstalkSources_1029425_9.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/6eb2923a-425d-4a36-b4e2-8121e105efc5_isrForCrosstalkSources_1029425_9.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2eb6b4864dd2a91f28b05fdf044e8c7a with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/942a37c5-6d27-480e-80eb-5dadcb203904_isrForCrosstalkSources_1029424_57.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/942a37c5-6d27-480e-80eb-5dadcb203904_isrForCrosstalkSources_1029424_57.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f5b9a0d693ec3da4f8ab5f078022e94b with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cc473ed9-e7ff-4745-8acb-38b1d503f736_isrForCrosstalkSources_1029424_1.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cc473ed9-e7ff-4745-8acb-38b1d503f736_isrForCrosstalkSources_1029424_1.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75af5e978a4784a4c85d05da5be86ed7 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/4408e0e5-400f-4a56-99a4-ca4b421597e4_isrForCrosstalkSources_1029418_29.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/4408e0e5-400f-4a56-99a4-ca4b421597e4_isrForCrosstalkSources_1029418_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:38 proc_lsst.multi:146 [INFO] found job 17113722 in provider astro proc_lsst.multi INFO: found job 17113722 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6a0209ccfe530774173f0ae1f2bd75c with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ae94d220-b028-4235-8b49-ab4c5a8bb8ec_isrForCrosstalkSources_1029424_19.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ae94d220-b028-4235-8b49-ab4c5a8bb8ec_isrForCrosstalkSources_1029424_19.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 548cfcec99b37198dd0c71e994c00476 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e3dd497f-6cd6-4a48-b610-fad86dfc2065_isrForCrosstalkSources_1029420_25.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e3dd497f-6cd6-4a48-b610-fad86dfc2065_isrForCrosstalkSources_1029420_25.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d86cea8625ae9a71785a3ac7ed7e854 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e4d2a9ad-3dfa-405f-a537-f984f3f7278f_isrForCrosstalkSources_1029424_59.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e4d2a9ad-3dfa-405f-a537-f984f3f7278f_isrForCrosstalkSources_1029424_59.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2bcf32f248c0116729dfefee5056d2 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0f014ff3-d809-4213-8bcb-d9c54348c240_isrForCrosstalkSources_1029419_15.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0f014ff3-d809-4213-8bcb-d9c54348c240_isrForCrosstalkSources_1029419_15.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcf3700df04cadd336024604904069ff with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9fa60787-8e60-4e34-9641-53bd9ffbb0f0_isrForCrosstalkSources_1029425_22.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9fa60787-8e60-4e34-9641-53bd9ffbb0f0_isrForCrosstalkSources_1029425_22.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66e7881dba7571086e750d730fe0c573 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f69a3fa2-b33f-430d-bcae-1f4075255a21_isrForCrosstalkSources_1029425_15.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f69a3fa2-b33f-430d-bcae-1f4075255a21_isrForCrosstalkSources_1029425_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:40 proc_lsst.multi:146 [INFO] found job 17113723 in provider astro proc_lsst.multi INFO: found job 17113723 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 848c2c2800130fbaa9b62c1cc619e656 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/aa685e13-e29d-4ef6-af6a-34442327e305_isrForCrosstalkSources_1029424_43.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/aa685e13-e29d-4ef6-af6a-34442327e305_isrForCrosstalkSources_1029424_43.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d33eff052dd10abdbbca8652586a0ec2 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3368ab94-a1ca-4008-a3b2-619e55037aaf_isrForCrosstalkSources_1029420_29.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3368ab94-a1ca-4008-a3b2-619e55037aaf_isrForCrosstalkSources_1029420_29.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce682f7d2a427c472ee8a2bf41e2b650 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fab5be19-8c28-4fcf-a20a-74a690711c2b_isrForCrosstalkSources_1029418_3.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fab5be19-8c28-4fcf-a20a-74a690711c2b_isrForCrosstalkSources_1029418_3.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74718830c7de5a9ccd8a2a66d57407cb with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3747afb4-5115-4c05-9c14-795276a13ac1_isrForCrosstalkSources_1029418_41.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3747afb4-5115-4c05-9c14-795276a13ac1_isrForCrosstalkSources_1029418_41.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61aa5b3b5a707c0d7859ebef067eb308 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d47c4c9e-f260-434a-b7ff-052ab5cff860_isrForCrosstalkSources_1029418_38.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d47c4c9e-f260-434a-b7ff-052ab5cff860_isrForCrosstalkSources_1029418_38.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67b62d12efbe3a2881df53f287dc6ca5 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/33ebdf42-bc11-4f2d-9d2c-71fc0ce07989_isrForCrosstalkSources_1029424_4.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/33ebdf42-bc11-4f2d-9d2c-71fc0ce07989_isrForCrosstalkSources_1029424_4.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f74cf483f34306c37b49216dd6128a with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/61d907db-b428-4138-ae22-41e50bd33952_isrForCrosstalkSources_1029418_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/61d907db-b428-4138-ae22-41e50bd33952_isrForCrosstalkSources_1029418_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:41 proc_lsst.multi:146 [INFO] found job 17113724 in provider astro proc_lsst.multi INFO: found job 17113724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 965e04282ebff7d103a3ea85daaaa194 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/80bfecde-4d22-4a03-886f-8cae45dc52fd_isrForCrosstalkSources_1029419_11.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/80bfecde-4d22-4a03-886f-8cae45dc52fd_isrForCrosstalkSources_1029419_11.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8356102e584a668243f4df7359784e4 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dfcf82a1-39fe-4e6c-8d3e-c46eacd8b79c_isrForCrosstalkSources_1029424_52.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dfcf82a1-39fe-4e6c-8d3e-c46eacd8b79c_isrForCrosstalkSources_1029424_52.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b97e95fcb84c55cd2a7edab66655e0c with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/55800fa1-5983-4d4d-8865-c115750008a8_isrForCrosstalkSources_1029419_31.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/55800fa1-5983-4d4d-8865-c115750008a8_isrForCrosstalkSources_1029419_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:43 proc_lsst.multi:146 [INFO] found job 17113725 in provider astro proc_lsst.multi INFO: found job 17113725 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 390632fd8fceff79a003f8aad798df38 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cc719f7a-3d30-47c9-a480-68247706bcbf_isrForCrosstalkSources_1029420_52.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cc719f7a-3d30-47c9-a480-68247706bcbf_isrForCrosstalkSources_1029420_52.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 030558ad3e787b5ca1db68411349e0a7 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e257b887-498d-4f74-a6e5-78daa3c48762_isrForCrosstalkSources_1029418_58.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e257b887-498d-4f74-a6e5-78daa3c48762_isrForCrosstalkSources_1029418_58.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc48a7ae03d79fc33e0b558eb4087b48 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/26355618-5c01-4fe8-a8c1-3fdad876e3f6_isrForCrosstalkSources_1029420_3.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/26355618-5c01-4fe8-a8c1-3fdad876e3f6_isrForCrosstalkSources_1029420_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:44 proc_lsst.multi:146 [INFO] found job 17113726 in provider astro proc_lsst.multi INFO: found job 17113726 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27a3708b8c663ee2e6d5b62440130802 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b9f491cf-526a-4b92-a954-8f6abfde9e7c_isrForCrosstalkSources_1029425_5.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b9f491cf-526a-4b92-a954-8f6abfde9e7c_isrForCrosstalkSources_1029425_5.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be297b785b2e41ad9f757d7dffbd7c0d with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/037eef55-556f-49fe-bd2d-6f614f64c357_isrForCrosstalkSources_1029426_27.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/037eef55-556f-49fe-bd2d-6f614f64c357_isrForCrosstalkSources_1029426_27.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a5aeb03796c10cb6df6c89cbc0eaf51 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/68555904-45c9-4672-a3af-df9f204e2f95_isrForCrosstalkSources_1029419_28.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/68555904-45c9-4672-a3af-df9f204e2f95_isrForCrosstalkSources_1029419_28.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68af3f22c04486a46c05845633b090e with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/b7c15585-24f8-4e14-b051-2c401f13a1bf_isrForCrosstalkSources_1029426_13.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/b7c15585-24f8-4e14-b051-2c401f13a1bf_isrForCrosstalkSources_1029426_13.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63d06db81e7312f23979f40a945c0148 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/da03c397-507d-4db3-a4eb-5f230bbb3cd1_isrForCrosstalkSources_1029418_10.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/da03c397-507d-4db3-a4eb-5f230bbb3cd1_isrForCrosstalkSources_1029418_10.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d453a8c39b93b783499d95258b98af9 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/58dda0bc-4f56-4cd0-921b-46202ce54c3f_isrForCrosstalkSources_1029425_39.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/58dda0bc-4f56-4cd0-921b-46202ce54c3f_isrForCrosstalkSources_1029425_39.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3ec860d32527899d731f94283c0d33b with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dc5e256b-b0a0-48df-aa00-10fc90cf9bcf_isrForCrosstalkSources_1029420_44.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dc5e256b-b0a0-48df-aa00-10fc90cf9bcf_isrForCrosstalkSources_1029420_44.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28321915e937b8f9cb7529b26b796fed with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/073bf926-ae4d-41f0-a8f0-a63a7703f3d1_isrForCrosstalkSources_1029420_22.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/073bf926-ae4d-41f0-a8f0-a63a7703f3d1_isrForCrosstalkSources_1029420_22.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c0474b20d6a380f589945fab397eaa0 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2601c431-f69e-43d9-a51c-33679c5d2482_isrForCrosstalkSources_1029425_25.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/2601c431-f69e-43d9-a51c-33679c5d2482_isrForCrosstalkSources_1029425_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:45 proc_lsst.multi:146 [INFO] found job 17113727 in provider astro proc_lsst.multi INFO: found job 17113727 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f69a36e51102ab85da78e00379161ee3 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/68148494-f6e6-4188-a674-59fce1f1d6fe_isrForCrosstalkSources_1029426_32.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/68148494-f6e6-4188-a674-59fce1f1d6fe_isrForCrosstalkSources_1029426_32.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2cf9d8c52761db36d37091c8fb52986 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/1e93adec-3b71-4756-9b63-074b8913809a_isrForCrosstalkSources_1029426_44.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/1e93adec-3b71-4756-9b63-074b8913809a_isrForCrosstalkSources_1029426_44.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6286dd8514ecfecfd6bab1a5923b1944 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/6ee58579-0902-4c1e-b918-d73fcdd42a44_isrForCrosstalkSources_1029426_24.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/6ee58579-0902-4c1e-b918-d73fcdd42a44_isrForCrosstalkSources_1029426_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:46 proc_lsst.multi:146 [INFO] found job 17113728 in provider astro proc_lsst.multi INFO: found job 17113728 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:47 proc_lsst.multi:146 [INFO] found job 17113729 in provider astro proc_lsst.multi INFO: found job 17113729 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5842960f03c3fc6123515f449e64e901 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/973b6ec9-f725-4add-97db-d32ff2282b7f_isrForCrosstalkSources_1029426_62.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/973b6ec9-f725-4add-97db-d32ff2282b7f_isrForCrosstalkSources_1029426_62.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ead592d4c4721ea8e961bb694c574f8d with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/048ec70e-102b-4faa-8ccd-03ecd443961c_isrForCrosstalkSources_1029425_3.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/048ec70e-102b-4faa-8ccd-03ecd443961c_isrForCrosstalkSources_1029425_3.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c453325a166cf53d1db34e4822b48d9f with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6f23d37d-9413-4c05-bf41-b6d43588db3b_isrForCrosstalkSources_1029420_27.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6f23d37d-9413-4c05-bf41-b6d43588db3b_isrForCrosstalkSources_1029420_27.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 850e0f5eee52f7dcbbdd049b25f703fc with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e3b88e82-8c59-479d-b16e-8db5e742dd6c_isrForCrosstalkSources_1029425_6.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e3b88e82-8c59-479d-b16e-8db5e742dd6c_isrForCrosstalkSources_1029425_6.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c94c3112066f5d41d958b8430a781fe with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/648cd1e8-f834-41ec-a088-208245634be0_isrForCrosstalkSources_1029426_53.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/648cd1e8-f834-41ec-a088-208245634be0_isrForCrosstalkSources_1029426_53.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59eaa32bb80c0cea50d7193dc5a94bd7 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0297a5c7-18f0-4d9c-81ec-5f44b6a9e682_isrForCrosstalkSources_1029424_47.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0297a5c7-18f0-4d9c-81ec-5f44b6a9e682_isrForCrosstalkSources_1029424_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:48 proc_lsst.multi:146 [INFO] found job 17113730 in provider astro proc_lsst.multi INFO: found job 17113730 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a163adef8cad9e311ac99985ddac575 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/82949259-5063-42e2-936d-85393203b425_isrForCrosstalkSources_1029424_54.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/82949259-5063-42e2-936d-85393203b425_isrForCrosstalkSources_1029424_54.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e659d1b106ec3b68106389caba47f4 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/817b68ed-3a48-48a4-9ed8-20f54421fe23_isrForCrosstalkSources_1029418_34.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/817b68ed-3a48-48a4-9ed8-20f54421fe23_isrForCrosstalkSources_1029418_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:48 proc_lsst.multi:146 [INFO] found job 17113731 in provider astro proc_lsst.multi INFO: found job 17113731 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 792d6aedd0aad1c96d6000d9d86daa66 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/0e76e83a-b9ef-4c14-9105-d47bdbb6cab5_isrForCrosstalkSources_1029425_59.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/0e76e83a-b9ef-4c14-9105-d47bdbb6cab5_isrForCrosstalkSources_1029425_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:49 proc_lsst.multi:146 [INFO] found job 17113736 in provider astro proc_lsst.multi INFO: found job 17113736 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31840cd4484c3e630e355797d6aae171 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/85da8d9e-35ec-4279-8ee1-beafd909278d_isrForCrosstalkSources_1029426_38.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/85da8d9e-35ec-4279-8ee1-beafd909278d_isrForCrosstalkSources_1029426_38.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36aae57f1329a99a22b5493d99ed138e with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2278a3ff-07fe-47a6-86d6-b0c850771e3d_isrForCrosstalkSources_1029426_48.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2278a3ff-07fe-47a6-86d6-b0c850771e3d_isrForCrosstalkSources_1029426_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:49 proc_lsst.multi:146 [INFO] found job 17113737 in provider astro proc_lsst.multi INFO: found job 17113737 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:50 proc_lsst.multi:146 [INFO] found job 17113738 in provider astro proc_lsst.multi INFO: found job 17113738 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:50 proc_lsst.multi:146 [INFO] found job 17113739 in provider astro proc_lsst.multi INFO: found job 17113739 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:51 proc_lsst.multi:146 [INFO] found job 17113740 in provider astro proc_lsst.multi INFO: found job 17113740 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:51 proc_lsst.multi:146 [INFO] found job 17113741 in provider astro proc_lsst.multi INFO: found job 17113741 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5289f51c165439c00cbe93c106fa2bc2 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/acfd2285-2e66-4879-ab53-f583396d3f7c_isrForCrosstalkSources_1029418_59.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/acfd2285-2e66-4879-ab53-f583396d3f7c_isrForCrosstalkSources_1029418_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:51 proc_lsst.multi:146 [INFO] found job 17113742 in provider astro proc_lsst.multi INFO: found job 17113742 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:52 proc_lsst.multi:146 [INFO] found job 17113743 in provider astro proc_lsst.multi INFO: found job 17113743 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:52 proc_lsst.multi:146 [INFO] found job 17113745 in provider astro proc_lsst.multi INFO: found job 17113745 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:52 proc_lsst.multi:146 [INFO] found job 17113746 in provider astro proc_lsst.multi INFO: found job 17113746 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:53 proc_lsst.multi:146 [INFO] found job 17113747 in provider astro proc_lsst.multi INFO: found job 17113747 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5991d6ff160ec49c532cb444b78b929e with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ccbb8877-c50f-45b2-9eb8-733f7aacbe50_isrForCrosstalkSources_1029426_8.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ccbb8877-c50f-45b2-9eb8-733f7aacbe50_isrForCrosstalkSources_1029426_8.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62fd06a98ad31c62425ff9a7ee02220c with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fe70c1ae-db99-4bda-a938-36c531aa0533_isrForCrosstalkSources_1029424_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fe70c1ae-db99-4bda-a938-36c531aa0533_isrForCrosstalkSources_1029424_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:55 proc_lsst.multi:146 [INFO] found job 17113748 in provider astro proc_lsst.multi INFO: found job 17113748 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 637ee98457d4553db55c3cad54ae861c with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/612c07a2-87a9-4c67-8194-88bd82e370c5_isrForCrosstalkSources_1029418_20.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/612c07a2-87a9-4c67-8194-88bd82e370c5_isrForCrosstalkSources_1029418_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:56 proc_lsst.multi:146 [INFO] found job 17113749 in provider astro proc_lsst.multi INFO: found job 17113749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb0892f0387c70b0558577e395ca341f with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/446633f8-364a-4925-8f22-3fd6f1654b1e_isrForCrosstalkSources_1029425_11.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/446633f8-364a-4925-8f22-3fd6f1654b1e_isrForCrosstalkSources_1029425_11.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea86d1c89baeae322eb3c24c0f6c1043 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/278a84f1-a260-4e6b-a174-3907dc3ad8a7_isrForCrosstalkSources_1029420_7.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/278a84f1-a260-4e6b-a174-3907dc3ad8a7_isrForCrosstalkSources_1029420_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:57 proc_lsst.multi:146 [INFO] found job 17113750 in provider astro proc_lsst.multi INFO: found job 17113750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfa1223f31e1b5a2a6a572ae3345d668 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3561db0-2e34-4c02-8c8d-3020f796b3e2_isrForCrosstalkSources_1029419_7.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3561db0-2e34-4c02-8c8d-3020f796b3e2_isrForCrosstalkSources_1029419_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:09:59 proc_lsst.multi:146 [INFO] found job 17113751 in provider astro proc_lsst.multi INFO: found job 17113751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7cd7b85c459bda66b012009e335ab82 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/061a1c50-8d50-423d-991a-fe7d15e701c0_isrForCrosstalkSources_1029425_7.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/061a1c50-8d50-423d-991a-fe7d15e701c0_isrForCrosstalkSources_1029425_7.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ff8db78199244ecf8970acace759f19 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8f5846cf-69df-4fdf-901e-f22b62c68cb9_isrForCrosstalkSources_1029424_34.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8f5846cf-69df-4fdf-901e-f22b62c68cb9_isrForCrosstalkSources_1029424_34.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a65f0e95602586c943b5677805540fbd with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/85df1d50-2b3b-4e03-ab67-4e3fb67c2bc7_isrForCrosstalkSources_1029425_16.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/85df1d50-2b3b-4e03-ab67-4e3fb67c2bc7_isrForCrosstalkSources_1029425_16.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9de0dfe264e766380c0cc05c5d16103e with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cc05929e-849c-4d5f-a1dd-ca7e5e23787e_isrForCrosstalkSources_1029425_32.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cc05929e-849c-4d5f-a1dd-ca7e5e23787e_isrForCrosstalkSources_1029425_32.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 381e8d95e92a1e4bc5372b412a717a1d with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7ec2ddf7-8ea1-4287-9d24-083e1024477f_isrForCrosstalkSources_1029425_24.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7ec2ddf7-8ea1-4287-9d24-083e1024477f_isrForCrosstalkSources_1029425_24.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6923415f953ff5573ceb5c1a951b0135 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9172c17f-ce74-471c-b8b9-6a2a160f52ae_isrForCrosstalkSources_1029425_19.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/9172c17f-ce74-471c-b8b9-6a2a160f52ae_isrForCrosstalkSources_1029425_19.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ba4683bf22efa0bed212b6874395ef1 with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c62c15da-a8ba-4ba5-aa92-04d5c3cd2ed8_isrForCrosstalkSources_1029425_21.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c62c15da-a8ba-4ba5-aa92-04d5c3cd2ed8_isrForCrosstalkSources_1029425_21.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 644b8d5b27c493d9b5025749a574b6c9 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1d930e6d-db74-4ba8-b667-ba527dfd2ea0_isrForCrosstalkSources_1029418_25.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1d930e6d-db74-4ba8-b667-ba527dfd2ea0_isrForCrosstalkSources_1029418_25.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c84697b883e29dec202d4c95feaadc32 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c66f8c10-73f9-4825-a674-2ecb23d34aa3_isrForCrosstalkSources_1029420_19.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c66f8c10-73f9-4825-a674-2ecb23d34aa3_isrForCrosstalkSources_1029420_19.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80144870f00404abba796855b8fe7599 with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c9dec7e7-6afc-4785-893d-1e768604affe_isrForCrosstalkSources_1029424_23.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c9dec7e7-6afc-4785-893d-1e768604affe_isrForCrosstalkSources_1029424_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:02 proc_lsst.multi:146 [INFO] found job 17113752 in provider astro proc_lsst.multi INFO: found job 17113752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2873c176a8f7bca3eebebbb598047e8 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/55536c30-7ed5-4dc8-a086-12a2517edddb_isrForCrosstalkSources_1029418_51.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/55536c30-7ed5-4dc8-a086-12a2517edddb_isrForCrosstalkSources_1029418_51.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac52ea6203823c0238385739e66cebb3 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b5aef3ca-5da6-4bdf-be6c-a2c621e7ac67_isrForCrosstalkSources_1029424_56.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/b5aef3ca-5da6-4bdf-be6c-a2c621e7ac67_isrForCrosstalkSources_1029424_56.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d3ecee5e92b398e94d1cc7c2a8b0689 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/2ebbab5a-5630-41f7-8764-0ef990dde462_isrForCrosstalkSources_1029420_4.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/2ebbab5a-5630-41f7-8764-0ef990dde462_isrForCrosstalkSources_1029420_4.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6038f43024e6d1a87a751bf82d2220da with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3983dce3-7641-4361-a3a6-07e85fca594a_isrForCrosstalkSources_1029426_9.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3983dce3-7641-4361-a3a6-07e85fca594a_isrForCrosstalkSources_1029426_9.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3901cb38ff2a75902194aa4cf4bc5b2e with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/426068b7-49e6-44f5-a496-d891fa7b14b3_isrForCrosstalkSources_1029426_7.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/426068b7-49e6-44f5-a496-d891fa7b14b3_isrForCrosstalkSources_1029426_7.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4859485edc325ed4003ae3637c3c4421 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/21d934c9-d4fb-40de-8508-d2581c3274f7_isrForCrosstalkSources_1029426_56.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/21d934c9-d4fb-40de-8508-d2581c3274f7_isrForCrosstalkSources_1029426_56.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e55167ac45b939143e230b21a8ae091a with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9caffb32-d24b-432a-bb42-88a533df3227_isrForCrosstalkSources_1029426_6.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/9caffb32-d24b-432a-bb42-88a533df3227_isrForCrosstalkSources_1029426_6.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4530b5521624a912e626a62797bbd98c with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/239f6494-d6c1-4bf7-af1b-474f12684bf4_isrForCrosstalkSources_1029426_5.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/239f6494-d6c1-4bf7-af1b-474f12684bf4_isrForCrosstalkSources_1029426_5.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ee720e49dd78a44737c28a54f02d4e4 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/956a6dee-d605-4218-8ed6-9ed9d348cae4_isrForCrosstalkSources_1029426_12.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/956a6dee-d605-4218-8ed6-9ed9d348cae4_isrForCrosstalkSources_1029426_12.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb40940c3c79a011d9b22392be300e4f with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10e5474b-ed65-4c46-9250-104410c50210_isrForCrosstalkSources_1029425_38.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10e5474b-ed65-4c46-9250-104410c50210_isrForCrosstalkSources_1029425_38.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23f4a7c801215efa160aaefbd7f15b78 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/71250836-353d-4a6f-8d63-959668de4059_isrForCrosstalkSources_1029424_29.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/71250836-353d-4a6f-8d63-959668de4059_isrForCrosstalkSources_1029424_29.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a720f000cfa4a649160b554b04f5fd0 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42dd7e60-410a-4a0c-875f-33da8abc2331_isrForCrosstalkSources_1029420_30.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42dd7e60-410a-4a0c-875f-33da8abc2331_isrForCrosstalkSources_1029420_30.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d7765a7c6a73575b91821453d5e4ee5 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac0dac1-a00c-4176-9345-f268247b9032_isrForCrosstalkSources_1029424_50.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac0dac1-a00c-4176-9345-f268247b9032_isrForCrosstalkSources_1029424_50.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 885d2fda9d883add05a258a0f21f92fb with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd92ce92-5628-4a9f-8d88-e7e8526eaeb4_isrForCrosstalkSources_1029424_61.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd92ce92-5628-4a9f-8d88-e7e8526eaeb4_isrForCrosstalkSources_1029424_61.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d31e58cdbcea1152e5fc7951e01c5d02 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd186c20-d5ee-4873-8260-57d91cf890ab_isrForCrosstalkSources_1029424_62.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/bd186c20-d5ee-4873-8260-57d91cf890ab_isrForCrosstalkSources_1029424_62.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8445525a0a21b63a96c50e0327306520 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d1c226fc-46fd-4921-a202-f4b71891db31_isrForCrosstalkSources_1029418_55.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/d1c226fc-46fd-4921-a202-f4b71891db31_isrForCrosstalkSources_1029418_55.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9adb29fe732dc8b91df674e4eee7fdb1 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/045e2be6-e221-46cf-8c3d-650c888ac517_isrForCrosstalkSources_1029418_31.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/045e2be6-e221-46cf-8c3d-650c888ac517_isrForCrosstalkSources_1029418_31.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2db9c768e3e6b52d185a8dae6a73d1ba with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c702fa64-05d6-455b-8494-ebc435ad48b1_isrForCrosstalkSources_1029418_6.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/c702fa64-05d6-455b-8494-ebc435ad48b1_isrForCrosstalkSources_1029418_6.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af94c09e81bd488a80649c2327781cc4 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8ba459a0-6e93-4b13-8424-a84bb7c20394_isrForCrosstalkSources_1029425_62.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8ba459a0-6e93-4b13-8424-a84bb7c20394_isrForCrosstalkSources_1029425_62.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a699df83657fa9cdd6d2497d6709cac with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/150515ed-2d44-49a4-b6e3-e0d80a24d751_isrForCrosstalkSources_1029420_46.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/150515ed-2d44-49a4-b6e3-e0d80a24d751_isrForCrosstalkSources_1029420_46.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38281ef56c5f3fa6c16bae54eedd6799 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4546d368-8f6c-4e02-b445-589ed5f46409_isrForCrosstalkSources_1029425_57.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/4546d368-8f6c-4e02-b445-589ed5f46409_isrForCrosstalkSources_1029425_57.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c7e0976da7f325aa84d32edb3abd52c with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/031c17ce-d930-4bb6-a3a8-057e7a732654_isrForCrosstalkSources_1029419_49.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/031c17ce-d930-4bb6-a3a8-057e7a732654_isrForCrosstalkSources_1029419_49.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 762e462e08f9eeef0cf00c377e3604b6 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/ca366cb1-6786-452f-a6e2-c9a4cc8c6204_isrForCrosstalkSources_1029419_58.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/ca366cb1-6786-452f-a6e2-c9a4cc8c6204_isrForCrosstalkSources_1029419_58.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87f9ceb7c8c38d8fc7bf2d120c39aa56 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a62aede1-3720-4703-af11-b406b3732432_isrForCrosstalkSources_1029425_52.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/a62aede1-3720-4703-af11-b406b3732432_isrForCrosstalkSources_1029425_52.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 529b7e5516668dffeb668822c046e457 with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/061c68bb-7226-49c2-8645-1e29d6c2a671_isrForCrosstalkSources_1029424_30.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/061c68bb-7226-49c2-8645-1e29d6c2a671_isrForCrosstalkSources_1029424_30.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc659892bcc21771ce9f6e0d1ca6ba8 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b4f7c228-038f-401e-8dd4-5eb928088b10_isrForCrosstalkSources_1029419_45.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b4f7c228-038f-401e-8dd4-5eb928088b10_isrForCrosstalkSources_1029419_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:04 proc_lsst.multi:146 [INFO] found job 17113754 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113754 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6d275e13cbeb67cf78ff9a562cc6196 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7c841969-a7e8-4951-b552-6aff6b4ab0fc_isrForCrosstalkSources_1029426_51.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/7c841969-a7e8-4951-b552-6aff6b4ab0fc_isrForCrosstalkSources_1029426_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:05 proc_lsst.multi:146 [INFO] found job 17113756 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113756 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:06 proc_lsst.multi:146 [INFO] found job 17113757 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113757 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db5e1fc28aecc0d39029e16a9fc8e145 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/70387920-83e9-421e-ac50-2e06f9c44390_isrForCrosstalkSources_1029425_17.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/70387920-83e9-421e-ac50-2e06f9c44390_isrForCrosstalkSources_1029425_17.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94392ac04bb9f1d77ddaeece2f3f82eb with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e6da8f24-3c51-4909-97e0-06da2cca8c19_isrForCrosstalkSources_1029419_9.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e6da8f24-3c51-4909-97e0-06da2cca8c19_isrForCrosstalkSources_1029419_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:07 proc_lsst.multi:146 [INFO] found job 17113763 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113763 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9a3c73d767d5224e314861de5b39b3ac with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1647661b-b141-488e-ac88-53b02e10d028_isrForCrosstalkSources_1029424_40.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/1647661b-b141-488e-ac88-53b02e10d028_isrForCrosstalkSources_1029424_40.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8ba6063be57f9fcaf2b2f95877695d0 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/dc0825fc-b06b-45ce-94b8-0daf36d5260c_isrForCrosstalkSources_1029419_36.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/dc0825fc-b06b-45ce-94b8-0daf36d5260c_isrForCrosstalkSources_1029419_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:07 proc_lsst.multi:146 [INFO] found job 17113764 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113764 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:08 proc_lsst.multi:146 [INFO] found job 17113766 in provider escience_a40 proc_lsst.multi INFO: found job 17113766 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:08 proc_lsst.multi:146 [INFO] found job 17113767 in provider escience_a40 proc_lsst.multi INFO: found job 17113767 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fbd503fe6fe3d6d0508dd4f3d979f24 with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/83b2d31f-ba00-42b9-b75f-624fe7c87ec0_isrForCrosstalkSources_1029426_43.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/83b2d31f-ba00-42b9-b75f-624fe7c87ec0_isrForCrosstalkSources_1029426_43.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d8b22b9cf9d0935db11944acb5218d6 with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42339ffc-c0b5-4322-a09c-2e8c44fc6373_isrForCrosstalkSources_1029420_47.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/42339ffc-c0b5-4322-a09c-2e8c44fc6373_isrForCrosstalkSources_1029420_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:10 proc_lsst.multi:146 [INFO] found job 17113768 in provider escience_a40 proc_lsst.multi INFO: found job 17113768 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:12 proc_lsst.multi:146 [INFO] found job 17113769 in provider escience_a40 proc_lsst.multi INFO: found job 17113769 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:12 proc_lsst.multi:146 [INFO] found job 17113770 in provider escience_a40 proc_lsst.multi INFO: found job 17113770 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bd67bbacc30a6675797c8285ea3e2b6 with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/279f827d-f34a-4bec-a8b3-637836d17db1_isrForCrosstalkSources_1029418_33.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/279f827d-f34a-4bec-a8b3-637836d17db1_isrForCrosstalkSources_1029418_33.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb95f95c28bbe90bafbacbd01a9a5d3c with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ddb0f8d4-4c41-4c6e-bfb2-21b7421e39a3_isrForCrosstalkSources_1029424_26.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ddb0f8d4-4c41-4c6e-bfb2-21b7421e39a3_isrForCrosstalkSources_1029424_26.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8437d4d4d194e039e7f7dad4e44d827 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/bf2f08e4-7ed7-4d0b-8ad4-9a00e9acd4b0_isrForCrosstalkSources_1029425_13.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/bf2f08e4-7ed7-4d0b-8ad4-9a00e9acd4b0_isrForCrosstalkSources_1029425_13.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 145163953fd635ef31e2d7a4c9a1f016 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/830e9542-fa9c-483f-882f-8cd3af576b44_isrForCrosstalkSources_1029419_19.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/830e9542-fa9c-483f-882f-8cd3af576b44_isrForCrosstalkSources_1029419_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:14 proc_lsst.multi:146 [INFO] found job 17113771 in provider escience_a40 proc_lsst.multi INFO: found job 17113771 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9739c6b3fa76c27f3e2efba4106a9eda with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1221ca07-df5b-485b-9d4b-5eb018ec7281_isrForCrosstalkSources_1029418_42.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/1221ca07-df5b-485b-9d4b-5eb018ec7281_isrForCrosstalkSources_1029418_42.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efddd838289f1dd9b154a36343d57ab9 with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/431f0b42-4701-425e-a8f6-606cb3b74296_isrForCrosstalkSources_1029426_23.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/431f0b42-4701-425e-a8f6-606cb3b74296_isrForCrosstalkSources_1029426_23.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 018a07923e51be1fcfa717dde0a467c0 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f19ab51b-02fc-4a37-a9b5-1fedc9268ba6_isrForCrosstalkSources_1029426_45.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f19ab51b-02fc-4a37-a9b5-1fedc9268ba6_isrForCrosstalkSources_1029426_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 42, active_tasks = 157 parsl.dataflow.strategy DEBUG: Executor multi has 157 active tasks, 42/0 running/pending blocks, and 43 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22745230885392 2024-03-16 11:10:16 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8e994759d0a2eecc9165116dbd3821 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87d0674e-939d-4a75-9fe0-94170a5091c8_isrForCrosstalkSources_1029424_10.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/87d0674e-939d-4a75-9fe0-94170a5091c8_isrForCrosstalkSources_1029424_10.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40b7daadd237e5679971ddb37eec1e3f with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd38674-80aa-4469-9e06-335d2bdec9f1_isrForCrosstalkSources_1029420_50.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fbd38674-80aa-4469-9e06-335d2bdec9f1_isrForCrosstalkSources_1029420_50.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6752b8aebdfc37bc92100ad8f60fa7b7 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3fdaba97-bb82-4f18-ba2b-42b422d76cca_isrForCrosstalkSources_1029420_59.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/3fdaba97-bb82-4f18-ba2b-42b422d76cca_isrForCrosstalkSources_1029420_59.stderr 2024-03-16 11:10:17 proc_lsst.multi:146 [INFO] found job 17113718 in provider astro proc_lsst.multi INFO: found job 17113718 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b4156d094836ca46aa8f6eacd4ecf48 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/aad1db82-540d-4cb8-8024-03f365588f0c_isrForCrosstalkSources_1029419_8.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/aad1db82-540d-4cb8-8024-03f365588f0c_isrForCrosstalkSources_1029419_8.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d4476e0f037505ed9c5b2fc6a11a7f4 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/53ff6062-cbb6-45b0-9754-e89f165357a0_isrForCrosstalkSources_1029419_21.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/53ff6062-cbb6-45b0-9754-e89f165357a0_isrForCrosstalkSources_1029419_21.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1294285c52311e89920ceddbfe10501 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/263345d7-a660-491b-a9ea-5dd83e6d6bfb_isrForCrosstalkSources_1029419_27.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/263345d7-a660-491b-a9ea-5dd83e6d6bfb_isrForCrosstalkSources_1029419_27.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6d5d2444a305bb3c44eea2707332c2f with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e2dc0924-7498-4be5-8bb1-de2b9c630d68_isrForCrosstalkSources_1029419_4.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e2dc0924-7498-4be5-8bb1-de2b9c630d68_isrForCrosstalkSources_1029419_4.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e79cc7b42662a144567a7b7ff4fc92ad with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d46fa904-816d-4aaf-b857-f1386b35df49_isrForCrosstalkSources_1029419_56.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d46fa904-816d-4aaf-b857-f1386b35df49_isrForCrosstalkSources_1029419_56.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95526e021316d1fa007a30b8b8e3d9e8 with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c233a2a0-3a40-48f6-87f4-5167386e2511_isrForCrosstalkSources_1029419_17.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c233a2a0-3a40-48f6-87f4-5167386e2511_isrForCrosstalkSources_1029419_17.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2600672ea6125028bac0048b8ba548d3 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/aeabf46f-d3a1-4a3f-926b-745f3d2ccb2d_isrForCrosstalkSources_1029425_45.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/aeabf46f-d3a1-4a3f-926b-745f3d2ccb2d_isrForCrosstalkSources_1029425_45.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb696bd7136597d1f3ae0cbb1471bb3b with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d864c404-963c-412a-a120-b4b1bce9236e_isrForCrosstalkSources_1029425_4.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d864c404-963c-412a-a120-b4b1bce9236e_isrForCrosstalkSources_1029425_4.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e205c7082ed227a2f32238d9f8cdc0f with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0be4b0a0-688e-486b-b04c-10d84c1e4216_isrForCrosstalkSources_1029424_25.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/0be4b0a0-688e-486b-b04c-10d84c1e4216_isrForCrosstalkSources_1029424_25.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b1baa0bffabaa92cafe85a6fda99a52 with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c4f6327f-0a61-4c0b-ae9e-dcb189f43468_isrForCrosstalkSources_1029424_22.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/c4f6327f-0a61-4c0b-ae9e-dcb189f43468_isrForCrosstalkSources_1029424_22.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1b967cde814224082aec1fe2278e7f1 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/50eb7eec-980f-4742-9ef6-93a7c8163af5_isrForCrosstalkSources_1029426_40.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/50eb7eec-980f-4742-9ef6-93a7c8163af5_isrForCrosstalkSources_1029426_40.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 541c38367b86577be1add94f9f2f2bb2 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3dacfb96-a670-4099-8c4a-3c4d36250014_isrForCrosstalkSources_1029426_39.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/3dacfb96-a670-4099-8c4a-3c4d36250014_isrForCrosstalkSources_1029426_39.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e33739a9deee970d18c52fb41a024288 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02b17ef7-1221-4edb-913f-f07c62cd260f_isrForCrosstalkSources_1029419_16.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02b17ef7-1221-4edb-913f-f07c62cd260f_isrForCrosstalkSources_1029419_16.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa9a9d1be621d1fb1eeeb86fa12ad9d6 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/55fc125b-a07c-42ba-961c-f9a26d246efd_isrForCrosstalkSources_1029424_46.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/55fc125b-a07c-42ba-961c-f9a26d246efd_isrForCrosstalkSources_1029424_46.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8384586d5a1716c2f3d58c58009e9e5 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f11b13c5-5795-4daf-a745-7b51d19da771_isrForCrosstalkSources_1029418_56.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f11b13c5-5795-4daf-a745-7b51d19da771_isrForCrosstalkSources_1029418_56.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3df1272e8f6b28baca995dafaca5ce5d with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c8b510c9-ebee-44a1-a02a-b7a476f04005_isrForCrosstalkSources_1029425_51.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/c8b510c9-ebee-44a1-a02a-b7a476f04005_isrForCrosstalkSources_1029425_51.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71367c206cc75e2f4e335dd8cd2e9f2e with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b8e14230-3c5a-4a58-8676-003cdeea3a21_isrForCrosstalkSources_1029418_39.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b8e14230-3c5a-4a58-8676-003cdeea3a21_isrForCrosstalkSources_1029418_39.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cb1246d8062d8e67076978b1b180a1b4 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7f06e93f-4a46-4932-9ae9-0ef9f4239b08_isrForCrosstalkSources_1029425_29.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7f06e93f-4a46-4932-9ae9-0ef9f4239b08_isrForCrosstalkSources_1029425_29.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fd1865e669e517d8c5122b930a6bd94 with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/1a3b5786-47e6-491e-ac22-861824b5f436_isrForCrosstalkSources_1029419_37.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/1a3b5786-47e6-491e-ac22-861824b5f436_isrForCrosstalkSources_1029419_37.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66b095a4fa66eb293d0d1b654e7916ba with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fb8d5e0b-0701-4ea6-901a-298bb8249f96_isrForCrosstalkSources_1029424_16.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/fb8d5e0b-0701-4ea6-901a-298bb8249f96_isrForCrosstalkSources_1029424_16.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493db03fa3b7b3269bfdf560c4ebc80c with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac744d3-f3f0-4a2d-a1d0-f94a4b3923bc_isrForCrosstalkSources_1029424_6.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/8ac744d3-f3f0-4a2d-a1d0-f94a4b3923bc_isrForCrosstalkSources_1029424_6.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 107a71d9571af56c2df006de70198853 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/33601bb5-d4ce-40ea-bc97-b901a15f2637_isrForCrosstalkSources_1029419_42.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/33601bb5-d4ce-40ea-bc97-b901a15f2637_isrForCrosstalkSources_1029419_42.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b80d7dad616dc8736fe93e32bbd78c0 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dd93becd-211d-48e3-940d-e05f2ed9adea_isrForCrosstalkSources_1029420_24.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/dd93becd-211d-48e3-940d-e05f2ed9adea_isrForCrosstalkSources_1029420_24.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aeb44f936d258b0ef128fa647707a0a5 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8fdcb722-413d-4943-b0d1-8892892cb2a4_isrForCrosstalkSources_1029425_54.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8fdcb722-413d-4943-b0d1-8892892cb2a4_isrForCrosstalkSources_1029425_54.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf41b1eef507d07096043293c535c600 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35d78006-e1c7-4264-bb73-c8e1b06bf1e1_isrForCrosstalkSources_1029425_60.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35d78006-e1c7-4264-bb73-c8e1b06bf1e1_isrForCrosstalkSources_1029425_60.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 072d17563f2ce6c7b34ff77018d4123b with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60ef00ac-658d-4846-b25b-80cb4ea06ad9_isrForCrosstalkSources_1029420_1.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60ef00ac-658d-4846-b25b-80cb4ea06ad9_isrForCrosstalkSources_1029420_1.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc950a34ef897926d85ea4b8be02adb9 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/9abcfcf6-e70d-49db-83aa-1ca1a0c575ca_isrForCrosstalkSources_1029420_61.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/9abcfcf6-e70d-49db-83aa-1ca1a0c575ca_isrForCrosstalkSources_1029420_61.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87d069dcafb61c0e8c4788eb4ca5b46f with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4bcdcd53-fd71-494b-875e-5ac3b56b52ec_isrForCrosstalkSources_1029426_41.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4bcdcd53-fd71-494b-875e-5ac3b56b52ec_isrForCrosstalkSources_1029426_41.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3183b4cb6a6bba99e529722632a215a3 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/5c7ebf9f-2ee4-4fb3-b549-817ad23cefa8_isrForCrosstalkSources_1029418_53.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/5c7ebf9f-2ee4-4fb3-b549-817ad23cefa8_isrForCrosstalkSources_1029418_53.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8af9b47aff58b41528acadcf55ff7e1e with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fc228add-e7bc-42a0-9893-7b2cb38fa033_isrForCrosstalkSources_1029420_48.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/fc228add-e7bc-42a0-9893-7b2cb38fa033_isrForCrosstalkSources_1029420_48.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c40ebc77e2674ddda8aac435ad3babb with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/54ce4dff-7aa4-433b-9904-013bfd1742ee_isrForCrosstalkSources_1029420_58.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/54ce4dff-7aa4-433b-9904-013bfd1742ee_isrForCrosstalkSources_1029420_58.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 642b6197eaeceec15d91c0d6f01dc62f with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/490da1ea-4bfb-4545-a009-25298ac38aa9_isrForCrosstalkSources_1029426_55.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/490da1ea-4bfb-4545-a009-25298ac38aa9_isrForCrosstalkSources_1029426_55.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a05f3acf63bcc4f6ad6c9b914682387a with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bf9d60fa-1590-4120-bc76-f98a0df87402_isrForCrosstalkSources_1029418_52.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/bf9d60fa-1590-4120-bc76-f98a0df87402_isrForCrosstalkSources_1029418_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:18 proc_lsst.multi:146 [INFO] found job 17113719 in provider astro proc_lsst.multi INFO: found job 17113719 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:19 proc_lsst.multi:146 [INFO] found job 17113720 in provider astro proc_lsst.multi INFO: found job 17113720 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:20 proc_lsst.multi:146 [INFO] found job 17113721 in provider astro proc_lsst.multi INFO: found job 17113721 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:21 proc_lsst.multi:146 [INFO] found job 17113722 in provider astro proc_lsst.multi INFO: found job 17113722 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:21 proc_lsst.multi:146 [INFO] found job 17113723 in provider astro proc_lsst.multi INFO: found job 17113723 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:22 proc_lsst.multi:146 [INFO] found job 17113724 in provider astro proc_lsst.multi INFO: found job 17113724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:22 proc_lsst.multi:146 [INFO] found job 17113725 in provider astro proc_lsst.multi INFO: found job 17113725 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:22 proc_lsst.multi:146 [INFO] found job 17113726 in provider astro proc_lsst.multi INFO: found job 17113726 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:23 proc_lsst.multi:146 [INFO] found job 17113727 in provider astro proc_lsst.multi INFO: found job 17113727 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38e12d4106b6a4c43eea719e2abe5bb2 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/345aeb0d-be8a-4109-9b07-58b84962054e_isrForCrosstalkSources_1029419_55.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/345aeb0d-be8a-4109-9b07-58b84962054e_isrForCrosstalkSources_1029419_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:23 proc_lsst.multi:146 [INFO] found job 17113728 in provider astro proc_lsst.multi INFO: found job 17113728 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:23 proc_lsst.multi:146 [INFO] found job 17113729 in provider astro proc_lsst.multi INFO: found job 17113729 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aa2d0a8a5ccb9dcb993ac1093b9c154 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fce9c334-d63b-4b76-99c5-ff0b1db0e9bb_isrForCrosstalkSources_1029418_4.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fce9c334-d63b-4b76-99c5-ff0b1db0e9bb_isrForCrosstalkSources_1029418_4.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbaf62acf291f3767608f65e8104c42d with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/41b88e24-b592-4f30-87bd-9c5fc8f68419_isrForCrosstalkSources_1029425_61.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/41b88e24-b592-4f30-87bd-9c5fc8f68419_isrForCrosstalkSources_1029425_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:24 proc_lsst.multi:146 [INFO] found job 17113730 in provider astro proc_lsst.multi INFO: found job 17113730 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:24 proc_lsst.multi:146 [INFO] found job 17113731 in provider astro proc_lsst.multi INFO: found job 17113731 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b9a02857bb2934c24dce340c17602c9 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8b10307b-ecc5-4394-bd34-d3c96e2cf906_isrForCrosstalkSources_1029425_14.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/8b10307b-ecc5-4394-bd34-d3c96e2cf906_isrForCrosstalkSources_1029425_14.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00667634f3c352210aca5fcad960f48d with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6a740928-c18a-4902-a6f5-fc5facc4dda3_isrForCrosstalkSources_1029420_11.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6a740928-c18a-4902-a6f5-fc5facc4dda3_isrForCrosstalkSources_1029420_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:25 proc_lsst.multi:146 [INFO] found job 17113736 in provider astro proc_lsst.multi INFO: found job 17113736 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56779056dad402dfceab0596e6ecdb2c with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/41057a1e-70b6-45c4-bd04-76f40cb36f81_isrForCrosstalkSources_1029420_12.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/41057a1e-70b6-45c4-bd04-76f40cb36f81_isrForCrosstalkSources_1029420_12.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2480f8f9847b3bb11e66ef685802bb04 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b29e5666-920a-4ab9-9b67-cbdcc87fcfba_isrForCrosstalkSources_1029418_35.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/b29e5666-920a-4ab9-9b67-cbdcc87fcfba_isrForCrosstalkSources_1029418_35.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4455802bcd03cc9a49198dfe7a727b3 with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd679752-11ce-48f8-bbd9-ca534d4c9633_isrForCrosstalkSources_1029425_44.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fd679752-11ce-48f8-bbd9-ca534d4c9633_isrForCrosstalkSources_1029425_44.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 972d4011fdb29c74e195d10fe2b875a5 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/695dd3f4-757e-4367-8ff8-8b0d35633a70_isrForCrosstalkSources_1029425_53.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/695dd3f4-757e-4367-8ff8-8b0d35633a70_isrForCrosstalkSources_1029425_53.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ad092c96e8e80294dfc92105c686dc0 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/822ad98a-f39f-40d8-a17a-c27b14fbaeef_isrForCrosstalkSources_1029425_55.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/822ad98a-f39f-40d8-a17a-c27b14fbaeef_isrForCrosstalkSources_1029425_55.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c830f5ade6510c515ffca6c790b5fd75 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/bc6f4b85-9887-4572-a413-0085a248120d_isrForCrosstalkSources_1029426_50.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/bc6f4b85-9887-4572-a413-0085a248120d_isrForCrosstalkSources_1029426_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:27 proc_lsst.multi:146 [INFO] found job 17113737 in provider astro proc_lsst.multi INFO: found job 17113737 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:28 proc_lsst.multi:146 [INFO] found job 17113738 in provider astro proc_lsst.multi INFO: found job 17113738 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:29 proc_lsst.multi:146 [INFO] found job 17113739 in provider astro proc_lsst.multi INFO: found job 17113739 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99dcfba50d3a9c6d005db76dc9a18c00 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/22d257ef-8056-4262-b13a-554c02e62aca_isrForCrosstalkSources_1029426_26.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/22d257ef-8056-4262-b13a-554c02e62aca_isrForCrosstalkSources_1029426_26.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9800675fa27af2aff98aa88a4364d261 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/66c1f960-12fb-44e7-ae46-2155cf85def0_isrForCrosstalkSources_1029426_19.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/66c1f960-12fb-44e7-ae46-2155cf85def0_isrForCrosstalkSources_1029426_19.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4256fc9d3f2e588a2339b93f8022fa with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/05897549-661a-4c9f-a5fe-7a3360b6c111_isrForCrosstalkSources_1029426_33.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/05897549-661a-4c9f-a5fe-7a3360b6c111_isrForCrosstalkSources_1029426_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:33 proc_lsst.multi:146 [INFO] found job 17113740 in provider astro proc_lsst.multi INFO: found job 17113740 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6559efb8859bf582958bb38ed4b06d0b with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ad784513-f34d-49f0-b7b8-0e73d85d9657_isrForCrosstalkSources_1029418_27.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ad784513-f34d-49f0-b7b8-0e73d85d9657_isrForCrosstalkSources_1029418_27.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82f6866ba56e36c9fdf6665c0afff9d3 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/78fdc670-4a80-45b3-915b-4d88bbf49026_isrForCrosstalkSources_1029420_14.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/78fdc670-4a80-45b3-915b-4d88bbf49026_isrForCrosstalkSources_1029420_14.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 596183c1e7c2bf84ae11fef209d998d3 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c82eb9ce-ce26-419f-9e44-bd85a7dd490f_isrForCrosstalkSources_1029420_20.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c82eb9ce-ce26-419f-9e44-bd85a7dd490f_isrForCrosstalkSources_1029420_20.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 310bb8c4bd25e5f35150d6421fa4988f with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cf2a4865-d6a3-49b9-9bee-460d2605b05e_isrForCrosstalkSources_1029425_8.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/cf2a4865-d6a3-49b9-9bee-460d2605b05e_isrForCrosstalkSources_1029425_8.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ed6db41907f70d00946784842ee8d27 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/38f38998-235f-4351-884d-6a41e29d83cb_isrForCrosstalkSources_1029418_14.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/38f38998-235f-4351-884d-6a41e29d83cb_isrForCrosstalkSources_1029418_14.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d937f3f55c688f3f916b8e8a70ace0a with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/73720cef-851a-482e-9ef7-7a8bd5321c90_isrForCrosstalkSources_1029419_10.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/73720cef-851a-482e-9ef7-7a8bd5321c90_isrForCrosstalkSources_1029419_10.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22684ad6245d8c2f123f162756ec0fe9 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7d2cf6f5-aaab-48a5-8e0f-5fe52ec910d2_isrForCrosstalkSources_1029425_10.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/7d2cf6f5-aaab-48a5-8e0f-5fe52ec910d2_isrForCrosstalkSources_1029425_10.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4fa515d341b11a903fa4ec59c21fb3 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e7761761-226a-4f85-80d5-05aadc403180_isrForCrosstalkSources_1029426_28.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e7761761-226a-4f85-80d5-05aadc403180_isrForCrosstalkSources_1029426_28.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bbbddddaa04485aba9c6e8e6121e6be with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c9135493-0aa5-4c9e-8ea0-94704babf0bd_isrForCrosstalkSources_1029426_36.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/c9135493-0aa5-4c9e-8ea0-94704babf0bd_isrForCrosstalkSources_1029426_36.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af02eceae962c6922a80ab081fc4dff3 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/96f70810-b041-437b-a9dc-2e344c482a38_isrForCrosstalkSources_1029424_5.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/96f70810-b041-437b-a9dc-2e344c482a38_isrForCrosstalkSources_1029424_5.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8527d511fe3e8a95da1d244af2f74b7 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/fb233a35-8f65-4d39-8b6c-f3f84c59fbd2_isrForCrosstalkSources_1029426_58.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/fb233a35-8f65-4d39-8b6c-f3f84c59fbd2_isrForCrosstalkSources_1029426_58.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b7f8727e505f975f7ee8b546a1afb10 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/12885372-057b-4d68-bd84-e2838f1592f8_isrForCrosstalkSources_1029426_29.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/12885372-057b-4d68-bd84-e2838f1592f8_isrForCrosstalkSources_1029426_29.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e4a558322825f1105cf968a9c1a4bb with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2251dce1-5b57-4b38-ad65-a9073308b444_isrForCrosstalkSources_1029418_9.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2251dce1-5b57-4b38-ad65-a9073308b444_isrForCrosstalkSources_1029418_9.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3faed67135bc0c2205e35503c62b1c8b with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/5a0b355d-40ed-4928-ae59-8697afb1ccc3_isrForCrosstalkSources_1029419_48.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/5a0b355d-40ed-4928-ae59-8697afb1ccc3_isrForCrosstalkSources_1029419_48.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a6a39617e4467edc29d58b6a06b50d3 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/edd99368-51e1-42a1-a22b-28ff57d9d8c8_isrForCrosstalkSources_1029419_41.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/edd99368-51e1-42a1-a22b-28ff57d9d8c8_isrForCrosstalkSources_1029419_41.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 717457c343c16df491409b0438d6e552 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3104852-068d-4c0d-9774-89ca246cd5be_isrForCrosstalkSources_1029419_60.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/b3104852-068d-4c0d-9774-89ca246cd5be_isrForCrosstalkSources_1029419_60.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f28ac833cf41603949d9851dff7e167f with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d8f4c6b6-8820-4566-ab66-86e422f8daac_isrForCrosstalkSources_1029419_54.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/d8f4c6b6-8820-4566-ab66-86e422f8daac_isrForCrosstalkSources_1029419_54.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a1e2a2dc2654e44e82d6f413b767a9f with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ba3f1610-3edb-47dc-b175-f800978e3ec5_isrForCrosstalkSources_1029426_57.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ba3f1610-3edb-47dc-b175-f800978e3ec5_isrForCrosstalkSources_1029426_57.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22d88f14013483c00f27cccdf45ee45a with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/66fd287b-ca8e-4c4e-931b-e2acdced2bd9_isrForCrosstalkSources_1029418_49.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/66fd287b-ca8e-4c4e-931b-e2acdced2bd9_isrForCrosstalkSources_1029418_49.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77b5866e79220202634e1bf9385e891f with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a838bd8b-e61e-4e86-b753-d079f79a39b0_isrForCrosstalkSources_1029420_5.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a838bd8b-e61e-4e86-b753-d079f79a39b0_isrForCrosstalkSources_1029420_5.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfcd4611bd78c54ebd5b3cfd56e189f1 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/55bb8c74-8d95-426a-90c5-85d05b6daad3_isrForCrosstalkSources_1029420_2.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/55bb8c74-8d95-426a-90c5-85d05b6daad3_isrForCrosstalkSources_1029420_2.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70bbd280eaed7c44835ea65d9fc8aeb with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6d390c1d-3979-496d-848a-bbc59bda5aef_isrForCrosstalkSources_1029420_41.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/6d390c1d-3979-496d-848a-bbc59bda5aef_isrForCrosstalkSources_1029420_41.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8564571fd15ffc501e74a68e89533ed1 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4baeae98-2c4c-4cc0-a5cd-140dccdb8816_isrForCrosstalkSources_1029426_59.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/4baeae98-2c4c-4cc0-a5cd-140dccdb8816_isrForCrosstalkSources_1029426_59.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e361ac2d3dc3d1cd58cf52198639bf6 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b0d27486-d5dd-423f-aa95-25002ad51c51_isrForCrosstalkSources_1029425_35.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/b0d27486-d5dd-423f-aa95-25002ad51c51_isrForCrosstalkSources_1029425_35.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6aef2f6ffe7faf6430389634bcb7bad with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cec7f866-b785-4101-b075-d72537188622_isrForCrosstalkSources_1029420_36.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/cec7f866-b785-4101-b075-d72537188622_isrForCrosstalkSources_1029420_36.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22db6e0d0b01b43f419d7ad749ab259a with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/618911b2-6a47-49b8-a72d-8933ed96906b_isrForCrosstalkSources_1029420_15.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/618911b2-6a47-49b8-a72d-8933ed96906b_isrForCrosstalkSources_1029420_15.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a60eb74e7a7737c1ebc540e0b4d6b955 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/08c87446-46d8-4403-a458-e0961bd92dd6_isrForCrosstalkSources_1029420_38.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/08c87446-46d8-4403-a458-e0961bd92dd6_isrForCrosstalkSources_1029420_38.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 222059865bd56a18fdfe838c6a8c1d12 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d8bd2247-161b-421d-b7ea-78fdbf7084b1_isrForCrosstalkSources_1029425_58.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/d8bd2247-161b-421d-b7ea-78fdbf7084b1_isrForCrosstalkSources_1029425_58.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d83f5448a9ee75245fdc894e45b30ce with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ea010178-4eac-41da-aa8b-ee9c383c7706_isrForCrosstalkSources_1029424_44.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/ea010178-4eac-41da-aa8b-ee9c383c7706_isrForCrosstalkSources_1029424_44.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd41c8dbc1a0d3e34b05be335ed5b496 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e061b789-e4b9-4e78-964a-bae20512b22f_isrForCrosstalkSources_1029420_54.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/e061b789-e4b9-4e78-964a-bae20512b22f_isrForCrosstalkSources_1029420_54.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5082366e492ea04026702e44ec25398 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/f216d3ca-1f19-4c89-bf9d-be2d4dbd8971_isrForCrosstalkSources_1029424_11.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/f216d3ca-1f19-4c89-bf9d-be2d4dbd8971_isrForCrosstalkSources_1029424_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:35 proc_lsst.multi:146 [INFO] found job 17113741 in provider astro proc_lsst.multi INFO: found job 17113741 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a74e1a26178c72d25fc4b78678e443c with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/05317ca0-01bc-4268-a5e8-fc2011528316_isrForCrosstalkSources_1029425_49.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/05317ca0-01bc-4268-a5e8-fc2011528316_isrForCrosstalkSources_1029425_49.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc2f6e13c5ebf01127040b22c628a27f with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c20804a6-b261-48fa-bc7b-3a52986e3ea7_isrForCrosstalkSources_1029419_33.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/c20804a6-b261-48fa-bc7b-3a52986e3ea7_isrForCrosstalkSources_1029419_33.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6403fe06d9583fd82ad86996c7b1ea86 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ba06b8a8-e8b2-4bdd-9935-5086a820176d_isrForCrosstalkSources_1029418_32.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/ba06b8a8-e8b2-4bdd-9935-5086a820176d_isrForCrosstalkSources_1029418_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:36 proc_lsst.multi:146 [INFO] found job 17113742 in provider astro proc_lsst.multi INFO: found job 17113742 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d80a0287cb36210cc48e882cd2a742 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/545d3c60-4bb4-4430-82a3-4937d36b9cfd_isrForCrosstalkSources_1029426_10.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/545d3c60-4bb4-4430-82a3-4937d36b9cfd_isrForCrosstalkSources_1029426_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:37 proc_lsst.multi:146 [INFO] found job 17113743 in provider astro proc_lsst.multi INFO: found job 17113743 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d67883443f88f38d420df573c3d8c207 with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/00cc46bd-9d37-4e72-8414-ff058746c301_isrForCrosstalkSources_1029425_40.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/00cc46bd-9d37-4e72-8414-ff058746c301_isrForCrosstalkSources_1029425_40.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1fdb94dbd97698953291732debf9bc7 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/13b71a85-744b-48ba-aecc-2aee2115f8b4_isrForCrosstalkSources_1029418_40.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/13b71a85-744b-48ba-aecc-2aee2115f8b4_isrForCrosstalkSources_1029418_40.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 924bba4c7c76f4bf0759b40f8edaa93a with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/44478e51-af13-41e4-8e08-142a6b455f5b_isrForCrosstalkSources_1029424_49.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/44478e51-af13-41e4-8e08-142a6b455f5b_isrForCrosstalkSources_1029424_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:38 proc_lsst.multi:146 [INFO] found job 17113745 in provider astro proc_lsst.multi INFO: found job 17113745 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:38 proc_lsst.multi:146 [INFO] found job 17113746 in provider astro proc_lsst.multi INFO: found job 17113746 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:39 proc_lsst.multi:146 [INFO] found job 17113747 in provider astro proc_lsst.multi INFO: found job 17113747 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 367a01c6958ae2ba041fe2ed6ac0e413 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/6a83c6cf-f001-4763-808d-3fb7598ddb67_isrForCrosstalkSources_1029424_53.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/6a83c6cf-f001-4763-808d-3fb7598ddb67_isrForCrosstalkSources_1029424_53.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d383bf7e93289e804c4ec03fb106fd7 with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3645be2f-be3a-4a46-a481-206445b9fc3d_isrForCrosstalkSources_1029418_60.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3645be2f-be3a-4a46-a481-206445b9fc3d_isrForCrosstalkSources_1029418_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:39 proc_lsst.multi:146 [INFO] found job 17113748 in provider astro proc_lsst.multi INFO: found job 17113748 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:39 proc_lsst.multi:146 [INFO] found job 17113749 in provider astro proc_lsst.multi INFO: found job 17113749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:40 proc_lsst.multi:146 [INFO] found job 17113750 in provider astro proc_lsst.multi INFO: found job 17113750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:40 proc_lsst.multi:146 [INFO] found job 17113751 in provider astro proc_lsst.multi INFO: found job 17113751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:41 proc_lsst.multi:146 [INFO] found job 17113752 in provider astro proc_lsst.multi INFO: found job 17113752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:41 proc_lsst.multi:146 [INFO] found job 17113754 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113754 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:43 proc_lsst.multi:146 [INFO] found job 17113756 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113756 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:44 proc_lsst.multi:146 [INFO] found job 17113757 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113757 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc3c3f75f31c4e7ddc23025fed0d8f00 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e02a003f-5731-43cd-a16e-92d50c3b6091_isrForCrosstalkSources_1029424_8.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/e02a003f-5731-43cd-a16e-92d50c3b6091_isrForCrosstalkSources_1029424_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:46 proc_lsst.multi:146 [INFO] found job 17113763 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113763 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c1bc923c871efaf92c82d9c2767661c with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/40ea9bb5-1ea6-4560-85ce-4abacaaf740f_isrForCrosstalkSources_1029426_3.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/40ea9bb5-1ea6-4560-85ce-4abacaaf740f_isrForCrosstalkSources_1029426_3.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01bb675c956e71f0326bcb0d83956dd8 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/593af60d-93b7-4cd7-99ca-e842f874a063_isrForCrosstalkSources_1029426_1.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/593af60d-93b7-4cd7-99ca-e842f874a063_isrForCrosstalkSources_1029426_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:48 proc_lsst.multi:146 [INFO] found job 17113764 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113764 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df4235947d1b9cc7e0937acdfc4634a with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/b8647834-42fe-4a38-b27f-7f683b72359a_isrForCrosstalkSources_1029420_18.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/b8647834-42fe-4a38-b27f-7f683b72359a_isrForCrosstalkSources_1029420_18.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18b9465712fd6490b4f41e6a3f3ab5ba with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8b719f93-59d4-40d4-a310-6dd5e17ae8ae_isrForCrosstalkSources_1029419_25.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/8b719f93-59d4-40d4-a310-6dd5e17ae8ae_isrForCrosstalkSources_1029419_25.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7ebd79baff9aa34f95d958989766819 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e24f7885-b983-4911-a9b0-9879083a46a8_isrForCrosstalkSources_1029425_33.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e24f7885-b983-4911-a9b0-9879083a46a8_isrForCrosstalkSources_1029425_33.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e298930e73dde0d68374624b55d58184 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/631e997a-fe90-4d1c-84b3-0304dab7d693_isrForCrosstalkSources_1029419_44.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/631e997a-fe90-4d1c-84b3-0304dab7d693_isrForCrosstalkSources_1029419_44.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4da7971515c3a2fc8c7acac613a61af0 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/052b4ac9-7e45-45a6-8f89-9527367bca5c_isrForCrosstalkSources_1029426_2.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/052b4ac9-7e45-45a6-8f89-9527367bca5c_isrForCrosstalkSources_1029426_2.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c3f814772448e363d6f485900af0358 with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/7794b13e-93dc-43da-ac70-b5f7c2ac2885_isrForCrosstalkSources_1029424_3.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/7794b13e-93dc-43da-ac70-b5f7c2ac2885_isrForCrosstalkSources_1029424_3.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd82d8f9fc5c9def2cfc50b0bea4de5 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/88aac6e1-15eb-4022-b179-b38c66046e33_isrForCrosstalkSources_1029419_59.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/88aac6e1-15eb-4022-b179-b38c66046e33_isrForCrosstalkSources_1029419_59.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 328734f6c9a0a6c3153ac2710d925c1b with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ea705ba9-90fa-4639-9dd0-5fd2473d84c4_isrForCrosstalkSources_1029426_14.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/ea705ba9-90fa-4639-9dd0-5fd2473d84c4_isrForCrosstalkSources_1029426_14.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9538118bc6394b6e72c7be5caabdfb04 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fb82a37d-b5e8-40d3-b0be-e71d2a8b6816_isrForCrosstalkSources_1029418_8.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/fb82a37d-b5e8-40d3-b0be-e71d2a8b6816_isrForCrosstalkSources_1029418_8.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5cc9bc81165b48f5aafc3ed0e5e5723 with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/d08b29f5-0d07-4302-b735-8fd0e234ff1f_isrForCrosstalkSources_1029426_25.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/d08b29f5-0d07-4302-b735-8fd0e234ff1f_isrForCrosstalkSources_1029426_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:50 proc_lsst.multi:146 [INFO] found job 17113766 in provider escience_a40 proc_lsst.multi INFO: found job 17113766 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 598f838efac11217641679bdbab0ac35 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/fbc64140-7af9-428a-8354-07c74e64e269_isrForCrosstalkSources_1029419_34.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/fbc64140-7af9-428a-8354-07c74e64e269_isrForCrosstalkSources_1029419_34.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11100903eebf8f9f2c25862ee5061abb with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/25b5d4b3-bb67-4064-8481-01da1ccbbc28_isrForCrosstalkSources_1029419_2.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/25b5d4b3-bb67-4064-8481-01da1ccbbc28_isrForCrosstalkSources_1029419_2.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbe1c78c5cc7986c31bbd2be4e29ceca with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/a95c6048-2305-469c-8eb9-4e6d79b0d9a7_isrForCrosstalkSources_1029419_39.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/a95c6048-2305-469c-8eb9-4e6d79b0d9a7_isrForCrosstalkSources_1029419_39.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743ba5fe2770225675c38d4be1c66aa0 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e395ab4f-4e87-4528-ba9b-373638569d44_isrForCrosstalkSources_1029426_22.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/e395ab4f-4e87-4528-ba9b-373638569d44_isrForCrosstalkSources_1029426_22.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fbfa08c0eef03a0f7b7eb0de1bdabeeb with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/69b8ed71-51fa-4062-832d-cfbf5cc623e9_isrForCrosstalkSources_1029419_22.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/69b8ed71-51fa-4062-832d-cfbf5cc623e9_isrForCrosstalkSources_1029419_22.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82e532d488cff2b855afc54f6f63bfa1 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/879b3c1d-7171-429f-8ce6-acc57a48f9de_isrForCrosstalkSources_1029420_32.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/879b3c1d-7171-429f-8ce6-acc57a48f9de_isrForCrosstalkSources_1029420_32.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ec00c04396efb8d9d1b45ba0b1d0aea with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e581a876-1b15-4530-b28b-a17723b6fda1_isrForCrosstalkSources_1029425_56.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e581a876-1b15-4530-b28b-a17723b6fda1_isrForCrosstalkSources_1029425_56.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a0300a9daa32de4e43062440e9f2cae with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0bf29add-f9a5-49fd-92e1-f6357fed2733_isrForCrosstalkSources_1029420_21.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0bf29add-f9a5-49fd-92e1-f6357fed2733_isrForCrosstalkSources_1029420_21.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ecfc369d6f24071ef350f56c3277786 with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/cbb5148c-819e-4e40-9445-7ef112b17552_isrForCrosstalkSources_1029419_12.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/cbb5148c-819e-4e40-9445-7ef112b17552_isrForCrosstalkSources_1029419_12.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a318e930ce32243d893dd77b254abec with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2c26aca3-21e1-4200-b5e2-9979a1895ea8_isrForCrosstalkSources_1029418_45.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/2c26aca3-21e1-4200-b5e2-9979a1895ea8_isrForCrosstalkSources_1029418_45.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bff30824dcb643fc0518502724bc65db with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/3fe939e2-e1bd-4207-b828-f8d735061ef8_isrForCrosstalkSources_1029425_46.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/3fe939e2-e1bd-4207-b828-f8d735061ef8_isrForCrosstalkSources_1029425_46.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b54ad6074d6be06772dcfe9dd2440155 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/4b0bf32a-ed46-40ce-8462-a93cf4bd60dc_isrForCrosstalkSources_1029420_51.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/4b0bf32a-ed46-40ce-8462-a93cf4bd60dc_isrForCrosstalkSources_1029420_51.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8d014ea7e81e3bc564748420459d72a with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60c78a95-4609-49e7-8255-01e4cbf03f9d_isrForCrosstalkSources_1029420_45.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/60c78a95-4609-49e7-8255-01e4cbf03f9d_isrForCrosstalkSources_1029420_45.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3721db98f3c3b71b2d3ceb6060dfe5c with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/305ebec3-2f6c-441e-85f8-9617f71b6dde_isrForCrosstalkSources_1029419_61.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/305ebec3-2f6c-441e-85f8-9617f71b6dde_isrForCrosstalkSources_1029419_61.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b57a6d0de07a9c24409b5c41de7f1e80 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/145449f9-dd98-4590-81b5-0140b82b9a54_isrForCrosstalkSources_1029426_37.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/145449f9-dd98-4590-81b5-0140b82b9a54_isrForCrosstalkSources_1029426_37.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3897daa72fba427a06d7141b9df5528 with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3cde1874-0bad-4a21-9196-97c1ab6285a3_isrForCrosstalkSources_1029424_17.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/3cde1874-0bad-4a21-9196-97c1ab6285a3_isrForCrosstalkSources_1029424_17.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339e5e152a12c184ff9bb841760f8d33 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dd148faa-2988-47e1-a101-5c432cc04803_isrForCrosstalkSources_1029424_28.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/dd148faa-2988-47e1-a101-5c432cc04803_isrForCrosstalkSources_1029424_28.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee7c281ce7789f51845f8c70ba6987eb with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e104756a-fea3-4501-b7b1-740ad98d56f1_isrForCrosstalkSources_1029419_62.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/e104756a-fea3-4501-b7b1-740ad98d56f1_isrForCrosstalkSources_1029419_62.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ba6c48672f128b66696defbd6d583a with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ce865e11-f8cf-46ee-a259-c66d75e67f09_isrForCrosstalkSources_1029425_2.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/ce865e11-f8cf-46ee-a259-c66d75e67f09_isrForCrosstalkSources_1029425_2.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c8c5c7e14a47c63d210b7a41f80250 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10f925ca-dd82-4426-8fe2-220f8f18d5ef_isrForCrosstalkSources_1029425_18.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/10f925ca-dd82-4426-8fe2-220f8f18d5ef_isrForCrosstalkSources_1029425_18.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e8ed857dda2b76726ca3c4fefd419d0 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c0f30630-b929-444a-bba8-1c3a833094a7_isrForCrosstalkSources_1029420_9.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/c0f30630-b929-444a-bba8-1c3a833094a7_isrForCrosstalkSources_1029420_9.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28c37dbd2b69e4dba49c2a2340dca0c4 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a5adb240-b651-4622-89ac-d235de5565d7_isrForCrosstalkSources_1029420_31.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a5adb240-b651-4622-89ac-d235de5565d7_isrForCrosstalkSources_1029420_31.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46d9f548fa08991f94ae7ca725b37213 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8d9cfd13-5e95-41d9-8d9a-2677657577f0_isrForCrosstalkSources_1029420_28.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8d9cfd13-5e95-41d9-8d9a-2677657577f0_isrForCrosstalkSources_1029420_28.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 069765522a41cfc4fae09e0a032c3e8d with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fe646c71-c40b-4296-9570-e9adb1d4fa51_isrForCrosstalkSources_1029425_42.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fe646c71-c40b-4296-9570-e9adb1d4fa51_isrForCrosstalkSources_1029425_42.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1bf278a172af437b0850c2fc1cf39aef with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8264ff65-7424-407b-9602-e45ca63d0ab6_isrForCrosstalkSources_1029420_62.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/8264ff65-7424-407b-9602-e45ca63d0ab6_isrForCrosstalkSources_1029420_62.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 616edaa5b69580a83edaaa57c0cb2fb7 with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f5a77c63-c803-4b7e-bc91-edda82a11acb_isrForCrosstalkSources_1029425_41.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/f5a77c63-c803-4b7e-bc91-edda82a11acb_isrForCrosstalkSources_1029425_41.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 980edb1e95ed5e02b210b7158d4fe528 with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3e34d722-dbf1-4ee6-b707-2c1c9a2dff76_isrForCrosstalkSources_1029418_30.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/3e34d722-dbf1-4ee6-b707-2c1c9a2dff76_isrForCrosstalkSources_1029418_30.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba277b4976b9be433764e55d2208ba9e with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/97a91374-548c-4e54-ae1f-0c888e1f642a_isrForCrosstalkSources_1029419_1.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/97a91374-548c-4e54-ae1f-0c888e1f642a_isrForCrosstalkSources_1029419_1.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1276cc5750770382bed8076b6b314860 with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e20fa402-d86b-4d8d-bbd8-d5989c76d217_isrForCrosstalkSources_1029418_44.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/e20fa402-d86b-4d8d-bbd8-d5989c76d217_isrForCrosstalkSources_1029418_44.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e91e3942e3139ed1c4a68bdca917548a with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/df54cab7-250e-4e37-9c2f-c57146686572_isrForCrosstalkSources_1029420_8.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/df54cab7-250e-4e37-9c2f-c57146686572_isrForCrosstalkSources_1029420_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:51 proc_lsst.multi:146 [INFO] found job 17113767 in provider escience_a40 proc_lsst.multi INFO: found job 17113767 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:51 proc_lsst.multi:146 [INFO] found job 17113768 in provider escience_a40 proc_lsst.multi INFO: found job 17113768 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:52 proc_lsst.multi:146 [INFO] found job 17113769 in provider escience_a40 proc_lsst.multi INFO: found job 17113769 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:52 proc_lsst.multi:146 [INFO] found job 17113770 in provider escience_a40 proc_lsst.multi INFO: found job 17113770 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:52 proc_lsst.multi:146 [INFO] found job 17113771 in provider escience_a40 proc_lsst.multi INFO: found job 17113771 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 42, active_tasks = 24 parsl.dataflow.strategy DEBUG: Executor multi has 24 active tasks, 42/0 running/pending blocks, and 43 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22745230885392 2024-03-16 11:10:53 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local 2024-03-16 11:10:53 proc_lsst.multi:146 [INFO] found job 17113718 in provider astro proc_lsst.multi INFO: found job 17113718 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:53 proc_lsst.multi:146 [INFO] found job 17113719 in provider astro proc_lsst.multi INFO: found job 17113719 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:53 proc_lsst.multi:146 [INFO] found job 17113720 in provider astro proc_lsst.multi INFO: found job 17113720 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:54 proc_lsst.multi:146 [INFO] found job 17113721 in provider astro proc_lsst.multi INFO: found job 17113721 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:54 proc_lsst.multi:146 [INFO] found job 17113722 in provider astro proc_lsst.multi INFO: found job 17113722 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:54 proc_lsst.multi:146 [INFO] found job 17113723 in provider astro proc_lsst.multi INFO: found job 17113723 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:55 proc_lsst.multi:146 [INFO] found job 17113724 in provider astro proc_lsst.multi INFO: found job 17113724 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:55 proc_lsst.multi:146 [INFO] found job 17113725 in provider astro proc_lsst.multi INFO: found job 17113725 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 8917a635c6cad56fe4e0a410000ebf49 with result from task 349 parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:55 proc_lsst.multi:146 [INFO] found job 17113726 in provider astro proc_lsst.multi INFO: found job 17113726 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a1b3056e-9497-4715-bf58-d2a4bdd586d3_isrForCrosstalkSources_1029420_33.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/a1b3056e-9497-4715-bf58-d2a4bdd586d3_isrForCrosstalkSources_1029420_33.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab892429e0c7274b139d49fc1db51372 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02ec3afd-5c36-47d7-a3ca-dbd1d6eeb8eb_isrForCrosstalkSources_1029419_24.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/02ec3afd-5c36-47d7-a3ca-dbd1d6eeb8eb_isrForCrosstalkSources_1029419_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:56 proc_lsst.multi:146 [INFO] found job 17113727 in provider astro proc_lsst.multi INFO: found job 17113727 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:56 proc_lsst.multi:146 [INFO] found job 17113728 in provider astro proc_lsst.multi INFO: found job 17113728 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d35b110ba34738cb18f2b26ebf73753 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0963b21a-2659-4690-87e5-3eb5d8494b25_isrForCrosstalkSources_1029419_23.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/0963b21a-2659-4690-87e5-3eb5d8494b25_isrForCrosstalkSources_1029419_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:57 proc_lsst.multi:146 [INFO] found job 17113729 in provider astro proc_lsst.multi INFO: found job 17113729 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:57 proc_lsst.multi:146 [INFO] found job 17113730 in provider astro proc_lsst.multi INFO: found job 17113730 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:58 proc_lsst.multi:146 [INFO] found job 17113731 in provider astro proc_lsst.multi INFO: found job 17113731 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2afbc13494ae7b0d966b3c4093cc05e4 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2696d6d4-8ae8-47bb-aac8-34ef0d2bb1e3_isrForCrosstalkSources_1029426_16.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/2696d6d4-8ae8-47bb-aac8-34ef0d2bb1e3_isrForCrosstalkSources_1029426_16.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e63b306d7d3ab1ae3b65eb7bd6431a49 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0c405e83-8b33-48a2-9c0e-4944cad5d2ad_isrForCrosstalkSources_1029420_10.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0c405e83-8b33-48a2-9c0e-4944cad5d2ad_isrForCrosstalkSources_1029420_10.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 618c64869606a42b907a879de4364599 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/239ff6a7-70f3-403a-99bb-b454c0f9f5dd_isrForCrosstalkSources_1029420_16.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/239ff6a7-70f3-403a-99bb-b454c0f9f5dd_isrForCrosstalkSources_1029420_16.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ebd63eb6bb9b4b3e874769e36f4e770 with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cde14b67-f0e2-4221-ac9f-c5eafd73466f_isrForCrosstalkSources_1029424_33.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/cde14b67-f0e2-4221-ac9f-c5eafd73466f_isrForCrosstalkSources_1029424_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:59 proc_lsst.multi:146 [INFO] found job 17113736 in provider astro proc_lsst.multi INFO: found job 17113736 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05cdbf722dedf53b512c2327827aaad5 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/dc007b7d-1344-459e-8ed1-0672aec17287_isrForCrosstalkSources_1029418_12.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/dc007b7d-1344-459e-8ed1-0672aec17287_isrForCrosstalkSources_1029418_12.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e250d0aedb9d58d779cbd883ba39152b with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/36b85304-6a90-4315-a211-9cc66288aeff_isrForCrosstalkSources_1029418_47.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/36b85304-6a90-4315-a211-9cc66288aeff_isrForCrosstalkSources_1029418_47.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 887f1d88ee9ff2e9d354b0150ec92619 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0a6d1ada-2153-47d4-a8b5-330f6f782920_isrForCrosstalkSources_1029420_60.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/0a6d1ada-2153-47d4-a8b5-330f6f782920_isrForCrosstalkSources_1029420_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:10:59 proc_lsst.multi:146 [INFO] found job 17113737 in provider astro proc_lsst.multi INFO: found job 17113737 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d7a9b9bd3cc53f3caf90a2de55db8ad with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/73517395-c784-4063-aaa5-3297bc17d4ba_isrForCrosstalkSources_1029420_49.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/73517395-c784-4063-aaa5-3297bc17d4ba_isrForCrosstalkSources_1029420_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:01 proc_lsst.multi:146 [INFO] found job 17113738 in provider astro proc_lsst.multi INFO: found job 17113738 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:02 proc_lsst.multi:146 [INFO] found job 17113739 in provider astro proc_lsst.multi INFO: found job 17113739 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:03 proc_lsst.multi:146 [INFO] found job 17113740 in provider astro proc_lsst.multi INFO: found job 17113740 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a45e65c93549e1c506b6b4e49a85e268 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/5284632f-5782-41c1-833b-ee342052d9d0_isrForCrosstalkSources_1029426_34.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/5284632f-5782-41c1-833b-ee342052d9d0_isrForCrosstalkSources_1029426_34.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eeacd49684d5571f801ec140ab67ffe with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f0b51f00-e6a9-4348-9ae1-05f6baacdd60_isrForCrosstalkSources_1029418_19.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/f0b51f00-e6a9-4348-9ae1-05f6baacdd60_isrForCrosstalkSources_1029418_19.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b84ec10ad3c73698c2528185b67cbe6 with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fb8468f7-6230-4492-8c37-c04cd2f83329_isrForCrosstalkSources_1029425_20.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/fb8468f7-6230-4492-8c37-c04cd2f83329_isrForCrosstalkSources_1029425_20.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e023e7b9a8decaeecdb2bec881fb9f4 with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/eb68501b-9e15-4607-896d-a141ff32bf76_isrForCrosstalkSources_1029420_39.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029420/eb68501b-9e15-4607-896d-a141ff32bf76_isrForCrosstalkSources_1029420_39.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d52f3e24ec9536829bd0f28f1b17c4 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a3f4d307-6573-45d5-8687-e81a20e1a91e_isrForCrosstalkSources_1029418_21.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/a3f4d307-6573-45d5-8687-e81a20e1a91e_isrForCrosstalkSources_1029418_21.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40253b1ee4410cee828cc17f40723cf0 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/93211b46-9f28-4919-856f-ea8a5919bef6_isrForCrosstalkSources_1029425_23.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/93211b46-9f28-4919-856f-ea8a5919bef6_isrForCrosstalkSources_1029425_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:03 proc_lsst.multi:146 [INFO] found job 17113741 in provider astro proc_lsst.multi INFO: found job 17113741 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f3a8b259f962ef7f5cbbb69ab12a8fb with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d96bca32-b596-4013-9553-24a4859e96f7_isrForCrosstalkSources_1029424_39.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029424/d96bca32-b596-4013-9553-24a4859e96f7_isrForCrosstalkSources_1029424_39.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 307eabfbfd9b722d4415f101a5916f4b with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/78a56626-9415-43d0-8799-4c43656c513b_isrForCrosstalkSources_1029418_7.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/78a56626-9415-43d0-8799-4c43656c513b_isrForCrosstalkSources_1029418_7.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a7a45bd19882252ae0b4dec7f4de26 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/77bd3f6a-59e1-4a3a-92ad-45a2484f7dd9_isrForCrosstalkSources_1029418_28.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029418/77bd3f6a-59e1-4a3a-92ad-45a2484f7dd9_isrForCrosstalkSources_1029418_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:04 proc_lsst.multi:146 [INFO] found job 17113742 in provider astro proc_lsst.multi INFO: found job 17113742 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8b7b958b88e0177c0c8aaabe919cfcb with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e9208400-0594-4a31-a38a-0a09de65de27_isrForCrosstalkSources_1029425_48.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/e9208400-0594-4a31-a38a-0a09de65de27_isrForCrosstalkSources_1029425_48.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c72ae38f52fc7fada2131ad37c3dbc07 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35c26f58-8c15-4769-8a86-3fd14129a481_isrForCrosstalkSources_1029425_47.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029425/35c26f58-8c15-4769-8a86-3fd14129a481_isrForCrosstalkSources_1029425_47.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7869fe9d2e86e509d4c20f3a69f042a6 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f9e5a183-b9b9-4536-8d80-137325078df0_isrForCrosstalkSources_1029426_31.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029426/f9e5a183-b9b9-4536-8d80-137325078df0_isrForCrosstalkSources_1029426_31.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7acea6a9f2919632e3e139e640cb6553 with result from task 370 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 372 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/180fe45b-bb35-4d30-9070-7161dedc4e6f_isrForCrosstalkSources_1029419_52.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/submit/DEEP/20210905/B1f/science#step0/20240316T110805Z/logs/isrForCrosstalkSources/1029419/180fe45b-bb35-4d30-9070-7161dedc4e6f_isrForCrosstalkSources_1029419_52.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:04 proc_lsst.multi:146 [INFO] found job 17113743 in provider astro proc_lsst.multi INFO: found job 17113743 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:05 proc_lsst.multi:146 [INFO] found job 17113745 in provider astro proc_lsst.multi INFO: found job 17113745 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:05 proc_lsst.multi:146 [INFO] found job 17113746 in provider astro proc_lsst.multi INFO: found job 17113746 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:05 proc_lsst.multi:146 [INFO] found job 17113747 in provider astro proc_lsst.multi INFO: found job 17113747 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:06 proc_lsst.multi:146 [INFO] found job 17113748 in provider astro proc_lsst.multi INFO: found job 17113748 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:06 proc_lsst.multi:146 [INFO] found job 17113749 in provider astro proc_lsst.multi INFO: found job 17113749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:06 proc_lsst.multi:146 [INFO] found job 17113750 in provider astro proc_lsst.multi INFO: found job 17113750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:06 proc_lsst.multi:146 [INFO] found job 17113751 in provider astro proc_lsst.multi INFO: found job 17113751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:07 proc_lsst.multi:146 [INFO] found job 17113752 in provider astro proc_lsst.multi INFO: found job 17113752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113718,17113719,17113720,17113721,17113722,17113723,17113724,17113725,17113726,17113727,17113728,17113729,17113730,17113731,17113736,17113737,17113738,17113739,17113740,17113741,17113742,17113743,17113745,17113746,17113747,17113748,17113749,17113750,17113751,17113752' parsl.providers.slurm.slurm DEBUG: squeue returned 17113752 R 17113751 R 17113749 R 17113750 R 17113747 R 17113748 R 17113746 R 17113745 R 17113742 R 17113743 R 17113741 R 17113738 R 17113739 R 17113740 R 17113736 R 17113737 R 17113730 R 17113731 R 17113726 R 17113727 R 17113728 R 17113729 R 17113720 R 17113721 R 17113722 R 17113723 R 17113724 R 17113725 R 17113718 R 17113719 R parsl.providers.slurm.slurm DEBUG: Updating job 17113752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113747 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113748 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113746 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113745 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113742 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113743 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113741 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113738 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113739 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113740 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113736 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113737 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113730 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113731 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113726 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113727 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113728 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113729 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113720 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113721 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113722 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113723 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113724 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113725 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113718 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113719 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:07 proc_lsst.multi:146 [INFO] found job 17113754 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113754 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:07 proc_lsst.multi:146 [INFO] found job 17113756 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113756 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:08 proc_lsst.multi:146 [INFO] found job 17113757 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113757 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:08 proc_lsst.multi:146 [INFO] found job 17113763 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113763 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:08 proc_lsst.multi:146 [INFO] found job 17113764 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113764 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113754,17113756,17113757,17113763,17113764' parsl.providers.slurm.slurm DEBUG: squeue returned 17113764 R 17113763 R 17113757 R 17113756 R 17113754 R parsl.providers.slurm.slurm DEBUG: Updating job 17113764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113754 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:09 proc_lsst.multi:146 [INFO] found job 17113766 in provider escience_a40 proc_lsst.multi INFO: found job 17113766 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:09 proc_lsst.multi:146 [INFO] found job 17113767 in provider escience_a40 proc_lsst.multi INFO: found job 17113767 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:09 proc_lsst.multi:146 [INFO] found job 17113768 in provider escience_a40 proc_lsst.multi INFO: found job 17113768 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:10 proc_lsst.multi:146 [INFO] found job 17113769 in provider escience_a40 proc_lsst.multi INFO: found job 17113769 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:10 proc_lsst.multi:146 [INFO] found job 17113770 in provider escience_a40 proc_lsst.multi INFO: found job 17113770 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING 2024-03-16 11:11:10 proc_lsst.multi:146 [INFO] found job 17113771 in provider escience_a40 proc_lsst.multi INFO: found job 17113771 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17113766,17113767,17113768,17113769,17113770,17113771' parsl.providers.slurm.slurm DEBUG: squeue returned 17113771 R 17113770 R 17113769 R 17113768 R 17113767 R 17113766 R parsl.providers.slurm.slurm DEBUG: Updating job 17113771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17113766 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 42, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 42/0 running/pending blocks, and 43 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (42) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 1.6689300537109375e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22745230885392 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=42, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 42 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25db2c52dff6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25db2c52dff6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9bd087cf3711 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9bd087cf3711 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f4e989a03d12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f4e989a03d12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8a3cad3fd4fd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8a3cad3fd4fd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 32d4cfe4345f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 32d4cfe4345f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c5baa9cd75f9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c5baa9cd75f9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1dd2b97b8c8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1dd2b97b8c8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f42785e851e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f42785e851e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4162f2ec1092 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4162f2ec1092 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a68f2fbcdef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a68f2fbcdef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9e896b4fa7fd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9e896b4fa7fd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: db647bb24ad4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: db647bb24ad4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43c8605cffb0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43c8605cffb0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 23641001ceb0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 23641001ceb0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f39bf2edca3b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f39bf2edca3b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e87e7bf0cd2e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e87e7bf0cd2e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8eda456d655c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8eda456d655c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e2d211c1edd7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e2d211c1edd7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7440ac0e7ef7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7440ac0e7ef7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c0fb201813d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c0fb201813d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3cf98a947719 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3cf98a947719 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52432da83edf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52432da83edf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6b30d7b0f8a0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6b30d7b0f8a0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28eba94acb94 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28eba94acb94 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61ef02eacd4f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61ef02eacd4f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5ef38e8b1a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5ef38e8b1a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a98424aeee1c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a98424aeee1c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c55399949c77 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c55399949c77 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6460646b7034 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6460646b7034 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8c521f4e3726 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8c521f4e3726 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fdc38cb212ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fdc38cb212ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0492e394f809 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0492e394f809 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e94169738ea0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e94169738ea0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf857c0954af parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf857c0954af parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 06b249dcbd4a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 06b249dcbd4a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e48c09450cb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e48c09450cb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c9d8860c8965 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c9d8860c8965 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdbbf3a936fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdbbf3a936fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 519a29072036 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 519a29072036 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3520acfdfd12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3520acfdfd12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 826ca3e73356 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 826ca3e73356 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4785bd69e3c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4785bd69e3c3 2024-03-16 11:11:11 proc_lsst.multi:146 [INFO] found job 17113737 in provider astro proc_lsst.multi INFO: found job 17113737 in provider astro 2024-03-16 11:11:11 proc_lsst.multi:201 [INFO] cancelling 17113737 on provider astro proc_lsst.multi INFO: cancelling 17113737 on provider astro 2024-03-16 11:11:11 proc_lsst.multi:146 [INFO] found job 17113728 in provider astro proc_lsst.multi INFO: found job 17113728 in provider astro 2024-03-16 11:11:11 proc_lsst.multi:201 [INFO] cancelling 17113728 on provider astro proc_lsst.multi INFO: cancelling 17113728 on provider astro 2024-03-16 11:11:11 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local 2024-03-16 11:11:11 proc_lsst.multi:201 [INFO] cancelling 15207 on provider local proc_lsst.multi INFO: cancelling 15207 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 15207 2024-03-16 11:11:11 proc_lsst.multi:146 [INFO] found job 17113723 in provider astro proc_lsst.multi INFO: found job 17113723 in provider astro 2024-03-16 11:11:11 proc_lsst.multi:201 [INFO] cancelling 17113723 on provider astro proc_lsst.multi INFO: cancelling 17113723 on provider astro 2024-03-16 11:11:12 proc_lsst.multi:146 [INFO] found job 17113718 in provider astro proc_lsst.multi INFO: found job 17113718 in provider astro 2024-03-16 11:11:12 proc_lsst.multi:201 [INFO] cancelling 17113718 on provider astro proc_lsst.multi INFO: cancelling 17113718 on provider astro 2024-03-16 11:11:12 proc_lsst.multi:146 [INFO] found job 17113749 in provider astro proc_lsst.multi INFO: found job 17113749 in provider astro 2024-03-16 11:11:12 proc_lsst.multi:201 [INFO] cancelling 17113749 on provider astro proc_lsst.multi INFO: cancelling 17113749 on provider astro 2024-03-16 11:11:12 proc_lsst.multi:146 [INFO] found job 17113731 in provider astro proc_lsst.multi INFO: found job 17113731 in provider astro 2024-03-16 11:11:12 proc_lsst.multi:201 [INFO] cancelling 17113731 on provider astro proc_lsst.multi INFO: cancelling 17113731 on provider astro 2024-03-16 11:11:12 proc_lsst.multi:146 [INFO] found job 17113745 in provider astro proc_lsst.multi INFO: found job 17113745 in provider astro 2024-03-16 11:11:12 proc_lsst.multi:201 [INFO] cancelling 17113745 on provider astro proc_lsst.multi INFO: cancelling 17113745 on provider astro 2024-03-16 11:11:13 proc_lsst.multi:146 [INFO] found job 17113742 in provider astro proc_lsst.multi INFO: found job 17113742 in provider astro 2024-03-16 11:11:13 proc_lsst.multi:201 [INFO] cancelling 17113742 on provider astro proc_lsst.multi INFO: cancelling 17113742 on provider astro 2024-03-16 11:11:14 proc_lsst.multi:146 [INFO] found job 17113729 in provider astro proc_lsst.multi INFO: found job 17113729 in provider astro 2024-03-16 11:11:14 proc_lsst.multi:201 [INFO] cancelling 17113729 on provider astro proc_lsst.multi INFO: cancelling 17113729 on provider astro 2024-03-16 11:11:14 proc_lsst.multi:146 [INFO] found job 17113720 in provider astro proc_lsst.multi INFO: found job 17113720 in provider astro 2024-03-16 11:11:14 proc_lsst.multi:201 [INFO] cancelling 17113720 on provider astro proc_lsst.multi INFO: cancelling 17113720 on provider astro 2024-03-16 11:11:15 proc_lsst.multi:146 [INFO] found job 17113743 in provider astro proc_lsst.multi INFO: found job 17113743 in provider astro 2024-03-16 11:11:15 proc_lsst.multi:201 [INFO] cancelling 17113743 on provider astro proc_lsst.multi INFO: cancelling 17113743 on provider astro 2024-03-16 11:11:15 proc_lsst.multi:146 [INFO] found job 17113750 in provider astro proc_lsst.multi INFO: found job 17113750 in provider astro 2024-03-16 11:11:15 proc_lsst.multi:201 [INFO] cancelling 17113750 on provider astro proc_lsst.multi INFO: cancelling 17113750 on provider astro 2024-03-16 11:11:15 proc_lsst.multi:146 [INFO] found job 17113757 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113757 in provider escience_rtx6k 2024-03-16 11:11:15 proc_lsst.multi:201 [INFO] cancelling 17113757 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113757 on provider escience_rtx6k 2024-03-16 11:11:15 proc_lsst.multi:146 [INFO] found job 17113767 in provider escience_a40 proc_lsst.multi INFO: found job 17113767 in provider escience_a40 2024-03-16 11:11:15 proc_lsst.multi:201 [INFO] cancelling 17113767 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113767 on provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:146 [INFO] found job 17113770 in provider escience_a40 proc_lsst.multi INFO: found job 17113770 in provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:201 [INFO] cancelling 17113770 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113770 on provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:146 [INFO] found job 17113771 in provider escience_a40 proc_lsst.multi INFO: found job 17113771 in provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:201 [INFO] cancelling 17113771 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113771 on provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:146 [INFO] found job 17113754 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113754 in provider escience_rtx6k 2024-03-16 11:11:16 proc_lsst.multi:201 [INFO] cancelling 17113754 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113754 on provider escience_rtx6k 2024-03-16 11:11:16 proc_lsst.multi:146 [INFO] found job 17113769 in provider escience_a40 proc_lsst.multi INFO: found job 17113769 in provider escience_a40 2024-03-16 11:11:16 proc_lsst.multi:201 [INFO] cancelling 17113769 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113769 on provider escience_a40 2024-03-16 11:11:17 proc_lsst.multi:146 [INFO] found job 17113766 in provider escience_a40 proc_lsst.multi INFO: found job 17113766 in provider escience_a40 2024-03-16 11:11:17 proc_lsst.multi:201 [INFO] cancelling 17113766 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113766 on provider escience_a40 2024-03-16 11:11:17 proc_lsst.multi:146 [INFO] found job 17113768 in provider escience_a40 proc_lsst.multi INFO: found job 17113768 in provider escience_a40 2024-03-16 11:11:17 proc_lsst.multi:201 [INFO] cancelling 17113768 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113768 on provider escience_a40 2024-03-16 11:11:17 proc_lsst.multi:146 [INFO] found job 17113763 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113763 in provider escience_rtx6k 2024-03-16 11:11:17 proc_lsst.multi:201 [INFO] cancelling 17113763 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113763 on provider escience_rtx6k 2024-03-16 11:11:17 proc_lsst.multi:146 [INFO] found job 17113756 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113756 in provider escience_rtx6k 2024-03-16 11:11:17 proc_lsst.multi:201 [INFO] cancelling 17113756 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113756 on provider escience_rtx6k 2024-03-16 11:11:18 proc_lsst.multi:146 [INFO] found job 17113764 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113764 in provider escience_rtx6k 2024-03-16 11:11:18 proc_lsst.multi:201 [INFO] cancelling 17113764 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113764 on provider escience_rtx6k 2024-03-16 11:11:18 proc_lsst.multi:146 [INFO] found job 17113739 in provider astro proc_lsst.multi INFO: found job 17113739 in provider astro 2024-03-16 11:11:18 proc_lsst.multi:201 [INFO] cancelling 17113739 on provider astro proc_lsst.multi INFO: cancelling 17113739 on provider astro 2024-03-16 11:11:18 proc_lsst.multi:146 [INFO] found job 17113719 in provider astro proc_lsst.multi INFO: found job 17113719 in provider astro 2024-03-16 11:11:18 proc_lsst.multi:201 [INFO] cancelling 17113719 on provider astro proc_lsst.multi INFO: cancelling 17113719 on provider astro 2024-03-16 11:11:18 proc_lsst.multi:146 [INFO] found job 17113724 in provider astro proc_lsst.multi INFO: found job 17113724 in provider astro 2024-03-16 11:11:18 proc_lsst.multi:201 [INFO] cancelling 17113724 on provider astro proc_lsst.multi INFO: cancelling 17113724 on provider astro 2024-03-16 11:11:19 proc_lsst.multi:146 [INFO] found job 17113722 in provider astro proc_lsst.multi INFO: found job 17113722 in provider astro 2024-03-16 11:11:19 proc_lsst.multi:201 [INFO] cancelling 17113722 on provider astro proc_lsst.multi INFO: cancelling 17113722 on provider astro 2024-03-16 11:11:19 proc_lsst.multi:146 [INFO] found job 17113741 in provider astro proc_lsst.multi INFO: found job 17113741 in provider astro 2024-03-16 11:11:19 proc_lsst.multi:201 [INFO] cancelling 17113741 on provider astro proc_lsst.multi INFO: cancelling 17113741 on provider astro 2024-03-16 11:11:19 proc_lsst.multi:146 [INFO] found job 17113738 in provider astro proc_lsst.multi INFO: found job 17113738 in provider astro 2024-03-16 11:11:19 proc_lsst.multi:201 [INFO] cancelling 17113738 on provider astro proc_lsst.multi INFO: cancelling 17113738 on provider astro 2024-03-16 11:11:19 proc_lsst.multi:146 [INFO] found job 17113752 in provider astro proc_lsst.multi INFO: found job 17113752 in provider astro 2024-03-16 11:11:19 proc_lsst.multi:201 [INFO] cancelling 17113752 on provider astro proc_lsst.multi INFO: cancelling 17113752 on provider astro 2024-03-16 11:11:20 proc_lsst.multi:146 [INFO] found job 17113746 in provider astro proc_lsst.multi INFO: found job 17113746 in provider astro 2024-03-16 11:11:20 proc_lsst.multi:201 [INFO] cancelling 17113746 on provider astro proc_lsst.multi INFO: cancelling 17113746 on provider astro 2024-03-16 11:11:20 proc_lsst.multi:146 [INFO] found job 17113726 in provider astro proc_lsst.multi INFO: found job 17113726 in provider astro 2024-03-16 11:11:20 proc_lsst.multi:201 [INFO] cancelling 17113726 on provider astro proc_lsst.multi INFO: cancelling 17113726 on provider astro 2024-03-16 11:11:20 proc_lsst.multi:146 [INFO] found job 17113721 in provider astro proc_lsst.multi INFO: found job 17113721 in provider astro 2024-03-16 11:11:20 proc_lsst.multi:201 [INFO] cancelling 17113721 on provider astro proc_lsst.multi INFO: cancelling 17113721 on provider astro 2024-03-16 11:11:20 proc_lsst.multi:146 [INFO] found job 17113748 in provider astro proc_lsst.multi INFO: found job 17113748 in provider astro 2024-03-16 11:11:20 proc_lsst.multi:201 [INFO] cancelling 17113748 on provider astro proc_lsst.multi INFO: cancelling 17113748 on provider astro 2024-03-16 11:11:21 proc_lsst.multi:146 [INFO] found job 17113725 in provider astro proc_lsst.multi INFO: found job 17113725 in provider astro 2024-03-16 11:11:21 proc_lsst.multi:201 [INFO] cancelling 17113725 on provider astro proc_lsst.multi INFO: cancelling 17113725 on provider astro 2024-03-16 11:11:21 proc_lsst.multi:146 [INFO] found job 17113751 in provider astro proc_lsst.multi INFO: found job 17113751 in provider astro 2024-03-16 11:11:21 proc_lsst.multi:201 [INFO] cancelling 17113751 on provider astro proc_lsst.multi INFO: cancelling 17113751 on provider astro 2024-03-16 11:11:21 proc_lsst.multi:146 [INFO] found job 17113727 in provider astro proc_lsst.multi INFO: found job 17113727 in provider astro 2024-03-16 11:11:21 proc_lsst.multi:201 [INFO] cancelling 17113727 on provider astro proc_lsst.multi INFO: cancelling 17113727 on provider astro 2024-03-16 11:11:21 proc_lsst.multi:146 [INFO] found job 17113736 in provider astro proc_lsst.multi INFO: found job 17113736 in provider astro 2024-03-16 11:11:21 proc_lsst.multi:201 [INFO] cancelling 17113736 on provider astro proc_lsst.multi INFO: cancelling 17113736 on provider astro 2024-03-16 11:11:22 proc_lsst.multi:146 [INFO] found job 17113747 in provider astro proc_lsst.multi INFO: found job 17113747 in provider astro 2024-03-16 11:11:22 proc_lsst.multi:201 [INFO] cancelling 17113747 on provider astro proc_lsst.multi INFO: cancelling 17113747 on provider astro 2024-03-16 11:11:22 proc_lsst.multi:146 [INFO] found job 17113740 in provider astro proc_lsst.multi INFO: found job 17113740 in provider astro 2024-03-16 11:11:22 proc_lsst.multi:201 [INFO] cancelling 17113740 on provider astro proc_lsst.multi INFO: cancelling 17113740 on provider astro 2024-03-16 11:11:22 proc_lsst.multi:146 [INFO] found job 17113730 in provider astro proc_lsst.multi INFO: found job 17113730 in provider astro 2024-03-16 11:11:22 proc_lsst.multi:201 [INFO] cancelling 17113730 on provider astro proc_lsst.multi INFO: cancelling 17113730 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-16 11:11:22 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-16 11:11:22 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17113718', '17113719', '17113720', '17113721', '17113722', '17113723', '17113724', '17113725', '17113726', '17113727', '17113728', '17113729', '17113730', '17113731', '17113736', '17113737', '17113738', '17113739', '17113740', '17113741', '17113742', '17113743', '17113745', '17113746', '17113747', '17113748', '17113749', '17113750', '17113751', '17113752', '17113766', '17113767', '17113768', '17113769', '17113770', '17113771', '17113754', '17113756', '17113757', '17113763', '17113764', '15207'] proc_lsst.multi INFO: new jobs since last cancel ['17113718', '17113719', '17113720', '17113721', '17113722', '17113723', '17113724', '17113725', '17113726', '17113727', '17113728', '17113729', '17113730', '17113731', '17113736', '17113737', '17113738', '17113739', '17113740', '17113741', '17113742', '17113743', '17113745', '17113746', '17113747', '17113748', '17113749', '17113750', '17113751', '17113752', '17113766', '17113767', '17113768', '17113769', '17113770', '17113771', '17113754', '17113756', '17113757', '17113763', '17113764', '15207'] 2024-03-16 11:11:22 proc_lsst.multi:146 [INFO] found job 17113718 in provider astro proc_lsst.multi INFO: found job 17113718 in provider astro 2024-03-16 11:11:22 proc_lsst.multi:201 [INFO] cancelling 17113718 on provider astro proc_lsst.multi INFO: cancelling 17113718 on provider astro 2024-03-16 11:11:22 proc_lsst.multi:146 [INFO] found job 17113719 in provider astro proc_lsst.multi INFO: found job 17113719 in provider astro 2024-03-16 11:11:22 proc_lsst.multi:201 [INFO] cancelling 17113719 on provider astro proc_lsst.multi INFO: cancelling 17113719 on provider astro 2024-03-16 11:11:23 proc_lsst.multi:146 [INFO] found job 17113720 in provider astro proc_lsst.multi INFO: found job 17113720 in provider astro 2024-03-16 11:11:23 proc_lsst.multi:201 [INFO] cancelling 17113720 on provider astro proc_lsst.multi INFO: cancelling 17113720 on provider astro 2024-03-16 11:11:23 proc_lsst.multi:146 [INFO] found job 17113721 in provider astro proc_lsst.multi INFO: found job 17113721 in provider astro 2024-03-16 11:11:23 proc_lsst.multi:201 [INFO] cancelling 17113721 on provider astro proc_lsst.multi INFO: cancelling 17113721 on provider astro 2024-03-16 11:11:23 proc_lsst.multi:146 [INFO] found job 17113722 in provider astro proc_lsst.multi INFO: found job 17113722 in provider astro 2024-03-16 11:11:23 proc_lsst.multi:201 [INFO] cancelling 17113722 on provider astro proc_lsst.multi INFO: cancelling 17113722 on provider astro 2024-03-16 11:11:23 proc_lsst.multi:146 [INFO] found job 17113723 in provider astro proc_lsst.multi INFO: found job 17113723 in provider astro 2024-03-16 11:11:23 proc_lsst.multi:201 [INFO] cancelling 17113723 on provider astro proc_lsst.multi INFO: cancelling 17113723 on provider astro 2024-03-16 11:11:24 proc_lsst.multi:146 [INFO] found job 17113724 in provider astro proc_lsst.multi INFO: found job 17113724 in provider astro 2024-03-16 11:11:24 proc_lsst.multi:201 [INFO] cancelling 17113724 on provider astro proc_lsst.multi INFO: cancelling 17113724 on provider astro 2024-03-16 11:11:24 proc_lsst.multi:146 [INFO] found job 17113725 in provider astro proc_lsst.multi INFO: found job 17113725 in provider astro 2024-03-16 11:11:24 proc_lsst.multi:201 [INFO] cancelling 17113725 on provider astro proc_lsst.multi INFO: cancelling 17113725 on provider astro 2024-03-16 11:11:24 proc_lsst.multi:146 [INFO] found job 17113726 in provider astro proc_lsst.multi INFO: found job 17113726 in provider astro 2024-03-16 11:11:24 proc_lsst.multi:201 [INFO] cancelling 17113726 on provider astro proc_lsst.multi INFO: cancelling 17113726 on provider astro 2024-03-16 11:11:24 proc_lsst.multi:146 [INFO] found job 17113727 in provider astro proc_lsst.multi INFO: found job 17113727 in provider astro 2024-03-16 11:11:24 proc_lsst.multi:201 [INFO] cancelling 17113727 on provider astro proc_lsst.multi INFO: cancelling 17113727 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113728 in provider astro proc_lsst.multi INFO: found job 17113728 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113728 on provider astro proc_lsst.multi INFO: cancelling 17113728 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113729 in provider astro proc_lsst.multi INFO: found job 17113729 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113729 on provider astro proc_lsst.multi INFO: cancelling 17113729 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113730 in provider astro proc_lsst.multi INFO: found job 17113730 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113730 on provider astro proc_lsst.multi INFO: cancelling 17113730 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113731 in provider astro proc_lsst.multi INFO: found job 17113731 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113731 on provider astro proc_lsst.multi INFO: cancelling 17113731 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113736 in provider astro proc_lsst.multi INFO: found job 17113736 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113736 on provider astro proc_lsst.multi INFO: cancelling 17113736 on provider astro 2024-03-16 11:11:25 proc_lsst.multi:146 [INFO] found job 17113737 in provider astro proc_lsst.multi INFO: found job 17113737 in provider astro 2024-03-16 11:11:25 proc_lsst.multi:201 [INFO] cancelling 17113737 on provider astro proc_lsst.multi INFO: cancelling 17113737 on provider astro 2024-03-16 11:11:26 proc_lsst.multi:146 [INFO] found job 17113738 in provider astro proc_lsst.multi INFO: found job 17113738 in provider astro 2024-03-16 11:11:26 proc_lsst.multi:201 [INFO] cancelling 17113738 on provider astro proc_lsst.multi INFO: cancelling 17113738 on provider astro 2024-03-16 11:11:26 proc_lsst.multi:146 [INFO] found job 17113739 in provider astro proc_lsst.multi INFO: found job 17113739 in provider astro 2024-03-16 11:11:26 proc_lsst.multi:201 [INFO] cancelling 17113739 on provider astro proc_lsst.multi INFO: cancelling 17113739 on provider astro 2024-03-16 11:11:26 proc_lsst.multi:146 [INFO] found job 17113740 in provider astro proc_lsst.multi INFO: found job 17113740 in provider astro 2024-03-16 11:11:26 proc_lsst.multi:201 [INFO] cancelling 17113740 on provider astro proc_lsst.multi INFO: cancelling 17113740 on provider astro 2024-03-16 11:11:26 proc_lsst.multi:146 [INFO] found job 17113741 in provider astro proc_lsst.multi INFO: found job 17113741 in provider astro 2024-03-16 11:11:26 proc_lsst.multi:201 [INFO] cancelling 17113741 on provider astro proc_lsst.multi INFO: cancelling 17113741 on provider astro 2024-03-16 11:11:26 proc_lsst.multi:146 [INFO] found job 17113742 in provider astro proc_lsst.multi INFO: found job 17113742 in provider astro 2024-03-16 11:11:26 proc_lsst.multi:201 [INFO] cancelling 17113742 on provider astro proc_lsst.multi INFO: cancelling 17113742 on provider astro 2024-03-16 11:11:27 proc_lsst.multi:146 [INFO] found job 17113743 in provider astro proc_lsst.multi INFO: found job 17113743 in provider astro 2024-03-16 11:11:27 proc_lsst.multi:201 [INFO] cancelling 17113743 on provider astro proc_lsst.multi INFO: cancelling 17113743 on provider astro 2024-03-16 11:11:27 proc_lsst.multi:146 [INFO] found job 17113745 in provider astro proc_lsst.multi INFO: found job 17113745 in provider astro 2024-03-16 11:11:27 proc_lsst.multi:201 [INFO] cancelling 17113745 on provider astro proc_lsst.multi INFO: cancelling 17113745 on provider astro 2024-03-16 11:11:27 proc_lsst.multi:146 [INFO] found job 17113746 in provider astro proc_lsst.multi INFO: found job 17113746 in provider astro 2024-03-16 11:11:27 proc_lsst.multi:201 [INFO] cancelling 17113746 on provider astro proc_lsst.multi INFO: cancelling 17113746 on provider astro 2024-03-16 11:11:27 proc_lsst.multi:146 [INFO] found job 17113747 in provider astro proc_lsst.multi INFO: found job 17113747 in provider astro 2024-03-16 11:11:27 proc_lsst.multi:201 [INFO] cancelling 17113747 on provider astro proc_lsst.multi INFO: cancelling 17113747 on provider astro 2024-03-16 11:11:28 proc_lsst.multi:146 [INFO] found job 17113748 in provider astro proc_lsst.multi INFO: found job 17113748 in provider astro 2024-03-16 11:11:28 proc_lsst.multi:201 [INFO] cancelling 17113748 on provider astro proc_lsst.multi INFO: cancelling 17113748 on provider astro 2024-03-16 11:11:28 proc_lsst.multi:146 [INFO] found job 17113749 in provider astro proc_lsst.multi INFO: found job 17113749 in provider astro 2024-03-16 11:11:28 proc_lsst.multi:201 [INFO] cancelling 17113749 on provider astro proc_lsst.multi INFO: cancelling 17113749 on provider astro 2024-03-16 11:11:28 proc_lsst.multi:146 [INFO] found job 17113750 in provider astro proc_lsst.multi INFO: found job 17113750 in provider astro 2024-03-16 11:11:28 proc_lsst.multi:201 [INFO] cancelling 17113750 on provider astro proc_lsst.multi INFO: cancelling 17113750 on provider astro 2024-03-16 11:11:28 proc_lsst.multi:146 [INFO] found job 17113751 in provider astro proc_lsst.multi INFO: found job 17113751 in provider astro 2024-03-16 11:11:28 proc_lsst.multi:201 [INFO] cancelling 17113751 on provider astro proc_lsst.multi INFO: cancelling 17113751 on provider astro 2024-03-16 11:11:28 proc_lsst.multi:146 [INFO] found job 17113752 in provider astro proc_lsst.multi INFO: found job 17113752 in provider astro 2024-03-16 11:11:28 proc_lsst.multi:201 [INFO] cancelling 17113752 on provider astro proc_lsst.multi INFO: cancelling 17113752 on provider astro 2024-03-16 11:11:29 proc_lsst.multi:146 [INFO] found job 17113766 in provider escience_a40 proc_lsst.multi INFO: found job 17113766 in provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:201 [INFO] cancelling 17113766 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113766 on provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:146 [INFO] found job 17113767 in provider escience_a40 proc_lsst.multi INFO: found job 17113767 in provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:201 [INFO] cancelling 17113767 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113767 on provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:146 [INFO] found job 17113768 in provider escience_a40 proc_lsst.multi INFO: found job 17113768 in provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:201 [INFO] cancelling 17113768 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113768 on provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:146 [INFO] found job 17113769 in provider escience_a40 proc_lsst.multi INFO: found job 17113769 in provider escience_a40 2024-03-16 11:11:29 proc_lsst.multi:201 [INFO] cancelling 17113769 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113769 on provider escience_a40 2024-03-16 11:11:30 proc_lsst.multi:146 [INFO] found job 17113770 in provider escience_a40 proc_lsst.multi INFO: found job 17113770 in provider escience_a40 2024-03-16 11:11:30 proc_lsst.multi:201 [INFO] cancelling 17113770 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113770 on provider escience_a40 2024-03-16 11:11:30 proc_lsst.multi:146 [INFO] found job 17113771 in provider escience_a40 proc_lsst.multi INFO: found job 17113771 in provider escience_a40 2024-03-16 11:11:30 proc_lsst.multi:201 [INFO] cancelling 17113771 on provider escience_a40 proc_lsst.multi INFO: cancelling 17113771 on provider escience_a40 2024-03-16 11:11:30 proc_lsst.multi:146 [INFO] found job 17113754 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113754 in provider escience_rtx6k 2024-03-16 11:11:30 proc_lsst.multi:201 [INFO] cancelling 17113754 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113754 on provider escience_rtx6k 2024-03-16 11:11:30 proc_lsst.multi:146 [INFO] found job 17113756 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113756 in provider escience_rtx6k 2024-03-16 11:11:30 proc_lsst.multi:201 [INFO] cancelling 17113756 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113756 on provider escience_rtx6k 2024-03-16 11:11:30 proc_lsst.multi:146 [INFO] found job 17113757 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113757 in provider escience_rtx6k 2024-03-16 11:11:30 proc_lsst.multi:201 [INFO] cancelling 17113757 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113757 on provider escience_rtx6k 2024-03-16 11:11:31 proc_lsst.multi:146 [INFO] found job 17113763 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113763 in provider escience_rtx6k 2024-03-16 11:11:31 proc_lsst.multi:201 [INFO] cancelling 17113763 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113763 on provider escience_rtx6k 2024-03-16 11:11:31 proc_lsst.multi:146 [INFO] found job 17113764 in provider escience_rtx6k proc_lsst.multi INFO: found job 17113764 in provider escience_rtx6k 2024-03-16 11:11:31 proc_lsst.multi:201 [INFO] cancelling 17113764 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17113764 on provider escience_rtx6k 2024-03-16 11:11:31 proc_lsst.multi:146 [INFO] found job 15207 in provider local proc_lsst.multi INFO: found job 15207 in provider local 2024-03-16 11:11:31 proc_lsst.multi:201 [INFO] cancelling 15207 on provider local proc_lsst.multi INFO: cancelling 15207 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 15207 parsl.providers.local.local WARNING: Failed to kill PID: 15207 and child processes on local 2024-03-16 11:11:32 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 197.6282 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210905_B1f_science#step0_20240316T110805Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 197.6363 seconds; current memory usage: 0.281 Gibyte, delta: 0.007 Gibyte, peak delta: 0.009 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 208.4718 seconds; current memory usage: 0.281 Gibyte, delta: 0.144 Gibyte, peak delta: 0.146 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.283 Gibyte (main), 0.281 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210905_B1f_science#step0_20240316T110805Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 3m30.539s user 0m37.852s sys 0m19.440s