+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210518/A1f --output-run DEEP/20210518/A1f/science#step0/20240316T000537Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.1914 seconds; current memory usage: 0.178 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.190 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0036 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0072 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/pipeline_orig.qgraph DEEP/20210518/A1f/science#step0/20240316T000537Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5351 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.6292 seconds; current memory usage: 0.272 Gibyte, delta: 0.093 Gibyte, peak delta: 0.082 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0100 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210518_A1f_science#step0_20240316T000537Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0480 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.1490 seconds; current memory usage: 0.273 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.263 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 83a6d1bf-9274-4960-9793-bafd10d3ca7e parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-16 00:05:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-16 00:05:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:05:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:05:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:05:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-16 00:05:59 proc_lsst.multi:170 [INFO] [multi] job_id 17843 proc_lsst.multi INFO: [multi] job_id 17843 2024-03-16 00:05:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:05:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:05:59 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 17843 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 17843 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 17843 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash f6c638398b6054333c01112778d47aae parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/cc2fa00f-fe47-48a3-bf5b-a6eacb7614b0_isrForCrosstalkSources_999144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/cc2fa00f-fe47-48a3-bf5b-a6eacb7614b0_isrForCrosstalkSources_999144_15.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash f9e5c9b49077a673e6f4730809a0f6cc parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/6609c52d-6e7d-4e91-8838-0a0f759c7b69_isrForCrosstalkSources_999142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/6609c52d-6e7d-4e91-8838-0a0f759c7b69_isrForCrosstalkSources_999142_40.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 9e7cfedbb48133bf3269e82b9e336bc0 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/08dd60c6-ba6a-41dd-b3d0-a5dfc181c720_isrForCrosstalkSources_999143_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/08dd60c6-ba6a-41dd-b3d0-a5dfc181c720_isrForCrosstalkSources_999143_26.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 6e73a1a40d2549283034c943b224c072 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a7de3386-8a5f-4dcb-874f-ab59d39d8d8e_isrForCrosstalkSources_999143_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a7de3386-8a5f-4dcb-874f-ab59d39d8d8e_isrForCrosstalkSources_999143_21.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash cc2feea0f0d62bfc8974201b5978f103 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe00a7c-17ee-4b8a-8f8d-9b03207cdf9b_isrForCrosstalkSources_999143_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe00a7c-17ee-4b8a-8f8d-9b03207cdf9b_isrForCrosstalkSources_999143_31.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 42a51478ab3aafc4731d134a24699090 parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31a4cb9d-38a2-4168-9caa-f4b1ab9276d0_isrForCrosstalkSources_999144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31a4cb9d-38a2-4168-9caa-f4b1ab9276d0_isrForCrosstalkSources_999144_21.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 235d634e7dfc122653591d660a42f29d parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/857494ed-2157-4e16-9073-ec534424314d_isrForCrosstalkSources_999144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/857494ed-2157-4e16-9073-ec534424314d_isrForCrosstalkSources_999144_54.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 0edfbcf57d22522d11291978855728aa parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b768824d-0760-451d-b813-c3cde9ca2b0d_isrForCrosstalkSources_999142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b768824d-0760-451d-b813-c3cde9ca2b0d_isrForCrosstalkSources_999142_18.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 801dbddda6c3c8b8e65cf0dc9b8f1914 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/25e6e757-ddcf-4849-99a6-22cf11f6cdd7_isrForCrosstalkSources_999142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/25e6e757-ddcf-4849-99a6-22cf11f6cdd7_isrForCrosstalkSources_999142_20.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash b912c00e7c5e25cf01294fa6c8956629 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0fbdb91-72a6-462e-9d64-a8e40bd67c91_isrForCrosstalkSources_999142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0fbdb91-72a6-462e-9d64-a8e40bd67c91_isrForCrosstalkSources_999142_22.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash e075ddfc701c57ab3acad3587fc7a4cb parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b967fee6-7f11-41df-afa9-ac931f9ff504_isrForCrosstalkSources_999142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b967fee6-7f11-41df-afa9-ac931f9ff504_isrForCrosstalkSources_999142_5.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash dd5a103400f01667ea627700a2f04135 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a0fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/56eb1580-9a7e-48b7-a134-61189f96ec5e_isrForCrosstalkSources_999142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/56eb1580-9a7e-48b7-a134-61189f96ec5e_isrForCrosstalkSources_999142_27.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash b026f5880885031ca5adf9a198da3989 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a500e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/33be11e6-cfb0-4eb9-b40a-a813f4bb6859_isrForCrosstalkSources_999144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/33be11e6-cfb0-4eb9-b40a-a813f4bb6859_isrForCrosstalkSources_999144_11.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash 57460911f83a3dc144ea37c57ce63adb parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/15620bd4-036a-4c40-95ac-b8aef504942a_isrForCrosstalkSources_999143_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/15620bd4-036a-4c40-95ac-b8aef504942a_isrForCrosstalkSources_999143_28.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash a49aedd0f2ef88d856c13742cd5af800 parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a505e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e0a54420-7a36-4fa3-8965-7354c7460ea3_isrForCrosstalkSources_999143_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e0a54420-7a36-4fa3-8965-7354c7460ea3_isrForCrosstalkSources_999143_2.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 1f7adc5384ffd44c023549b17763bf3f parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a502c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a0fe53bd-435a-4f09-9403-89b19d26bdc3_isrForCrosstalkSources_999144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a0fe53bd-435a-4f09-9403-89b19d26bdc3_isrForCrosstalkSources_999144_30.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash d5efb540a52d957b2c4cc1db1871cfba parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/49d4ef39-434a-4a00-b5d4-07a5271d8e20_isrForCrosstalkSources_999143_49.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/49d4ef39-434a-4a00-b5d4-07a5271d8e20_isrForCrosstalkSources_999143_49.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 1c4dea592c69185c05e20a3a2e496310 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/3cd35212-59a5-4a18-bc1c-13c2fad8f167_isrForCrosstalkSources_999142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/3cd35212-59a5-4a18-bc1c-13c2fad8f167_isrForCrosstalkSources_999142_34.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash babe608a3b5706e2704950a3274a997b parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d9b85a17-d257-47fa-8f3b-4daee87dd233_isrForCrosstalkSources_999143_39.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d9b85a17-d257-47fa-8f3b-4daee87dd233_isrForCrosstalkSources_999143_39.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash fcae3db2b9562f7176de59eb7cb4413c parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8c36fe52-b82a-4f15-a887-f76db2c7f36f_isrForCrosstalkSources_999143_57.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8c36fe52-b82a-4f15-a887-f76db2c7f36f_isrForCrosstalkSources_999143_57.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash 5ee2a49ba27ba95a47e014111dde7c6d parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/51dbd366-5440-43ed-a053-7cd9910072c0_isrForCrosstalkSources_999142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/51dbd366-5440-43ed-a053-7cd9910072c0_isrForCrosstalkSources_999142_52.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 4f6927c4bcf6bda94c72e665280dbbf2 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/dde246f8-4246-468c-849e-122f0e5993a6_isrForCrosstalkSources_999143_15.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/dde246f8-4246-468c-849e-122f0e5993a6_isrForCrosstalkSources_999143_15.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 9c063afe39960822f5ba9b32eba05c5e parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/b90c1139-7970-4151-ad50-43af8ecda18e_isrForCrosstalkSources_999144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/b90c1139-7970-4151-ad50-43af8ecda18e_isrForCrosstalkSources_999144_62.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 335df7f5ef319c800a4c209a6b0591ca parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3dac1764-961f-4676-863e-27504defdc99_isrForCrosstalkSources_999143_11.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3dac1764-961f-4676-863e-27504defdc99_isrForCrosstalkSources_999143_11.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash 21e6078b0bcc22fe9a46fb627e4565eb parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5c129f7-5862-41d0-8343-0bbe04c0b19c_isrForCrosstalkSources_999142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5c129f7-5862-41d0-8343-0bbe04c0b19c_isrForCrosstalkSources_999142_13.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 3b53d414a3c1153487d382596e29deca parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/01a02d87-8b4c-434b-8f40-0b5268cf4e02_isrForCrosstalkSources_999142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/01a02d87-8b4c-434b-8f40-0b5268cf4e02_isrForCrosstalkSources_999142_7.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 5d2bb5eccb951c56327724b9a186b129 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f33dd437-13b5-4dff-a9b5-890458a5fc33_isrForCrosstalkSources_999143_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f33dd437-13b5-4dff-a9b5-890458a5fc33_isrForCrosstalkSources_999143_9.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash b202080ae0ee2adcc6a7444881c09f28 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/143e63f9-d122-41b4-8d10-f2d2dcc734ca_isrForCrosstalkSources_999142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/143e63f9-d122-41b4-8d10-f2d2dcc734ca_isrForCrosstalkSources_999142_53.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 67eaa0a1b5c515a10c3521bb6432ea34 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b86d2ac3-d0c1-4825-aba1-6fadb7373918_isrForCrosstalkSources_999142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b86d2ac3-d0c1-4825-aba1-6fadb7373918_isrForCrosstalkSources_999142_44.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 7a82ee1e185c12c4538dbdaa9dde9a2c parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a2bee8f4-b720-4b38-9086-e35e8b68f688_isrForCrosstalkSources_999143_56.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a2bee8f4-b720-4b38-9086-e35e8b68f688_isrForCrosstalkSources_999143_56.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 8f9e02bd4aa01ddf13d913461fcfda78 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/403878c4-5f03-40d5-bfb3-37782c514db6_isrForCrosstalkSources_999143_10.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/403878c4-5f03-40d5-bfb3-37782c514db6_isrForCrosstalkSources_999143_10.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 2910b5eaf359d5045073c0ed1fedaf46 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31906b93-8b7a-43f7-9caf-1b0ff79d2aad_isrForCrosstalkSources_999144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31906b93-8b7a-43f7-9caf-1b0ff79d2aad_isrForCrosstalkSources_999144_51.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 2c387947cc37abce5a0054e803784b9d parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f0e8c79b-40c8-49c6-8989-b61826ad860b_isrForCrosstalkSources_999144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f0e8c79b-40c8-49c6-8989-b61826ad860b_isrForCrosstalkSources_999144_5.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 263d511a2e6ee1cdb4b91a3d490497fc parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/fd48f427-fea5-44ab-95ab-f77566137ce2_isrForCrosstalkSources_999144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/fd48f427-fea5-44ab-95ab-f77566137ce2_isrForCrosstalkSources_999144_28.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash bf8bfe52c485111d2d819068b47fa760 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02b66c12-9050-4bb2-be20-c82db2009cea_isrForCrosstalkSources_999142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02b66c12-9050-4bb2-be20-c82db2009cea_isrForCrosstalkSources_999142_51.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 1b6c66e4f51512e189ba94815ccc2c21 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5990d00c-d4fc-4112-b2dd-c9811f14b5dd_isrForCrosstalkSources_999144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5990d00c-d4fc-4112-b2dd-c9811f14b5dd_isrForCrosstalkSources_999144_8.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash f113335ba7c0f05b742387621668c287 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4c826a04-e794-4f8b-b1f2-9636555f2e19_isrForCrosstalkSources_999143_41.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4c826a04-e794-4f8b-b1f2-9636555f2e19_isrForCrosstalkSources_999143_41.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 5417ac050a3bf20d0ca8c21dd989ecfe parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/05957d3f-683a-4893-b8aa-e1eec262a5ee_isrForCrosstalkSources_999144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/05957d3f-683a-4893-b8aa-e1eec262a5ee_isrForCrosstalkSources_999144_49.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 3be046635b01f17b8a26a9f3b4226180 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/849743a7-3df1-4ac3-94cb-33a08202bc51_isrForCrosstalkSources_999142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/849743a7-3df1-4ac3-94cb-33a08202bc51_isrForCrosstalkSources_999142_54.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash fdbccdfe90689a9a104f08aa58bd5e3b parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9e3048ac-2a66-4bdc-a146-2e6cafc45787_isrForCrosstalkSources_999142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9e3048ac-2a66-4bdc-a146-2e6cafc45787_isrForCrosstalkSources_999142_11.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash c6786977a5b8ec9308168c2280c04c48 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ef2316d1-f4f9-4e64-b721-8a0ff7c9f934_isrForCrosstalkSources_999143_6.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ef2316d1-f4f9-4e64-b721-8a0ff7c9f934_isrForCrosstalkSources_999143_6.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash b5a2d15f92e3a143070c1694e7aded33 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8b4457f0-dbcc-47a2-a739-29a43badd961_isrForCrosstalkSources_999144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8b4457f0-dbcc-47a2-a739-29a43badd961_isrForCrosstalkSources_999144_14.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 9f665d306bcc23c3e32f9feaaf3035e8 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d0cfb7cd-c77a-4614-9ea2-030e0b3205c1_isrForCrosstalkSources_999142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d0cfb7cd-c77a-4614-9ea2-030e0b3205c1_isrForCrosstalkSources_999142_59.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 3976d510f8d0cb98c30a365490edb3ea parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/481877b2-f270-4bc2-9944-45bb66439675_isrForCrosstalkSources_999144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/481877b2-f270-4bc2-9944-45bb66439675_isrForCrosstalkSources_999144_58.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash df8916fad07b5c6ba6df7c50b34ddd47 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b804d082-8038-4156-bbc1-9e62d9ab8fbd_isrForCrosstalkSources_999142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b804d082-8038-4156-bbc1-9e62d9ab8fbd_isrForCrosstalkSources_999142_47.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash d4e6ad2704700968bcee683aa104a115 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/132f5606-5df2-4b3e-9c2e-b24043f260aa_isrForCrosstalkSources_999142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/132f5606-5df2-4b3e-9c2e-b24043f260aa_isrForCrosstalkSources_999142_29.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 1ba997bb29db2c69751e4fe308163e3d parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5c786791-767c-46fc-9fa3-88e6e60bc4a8_isrForCrosstalkSources_999142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5c786791-767c-46fc-9fa3-88e6e60bc4a8_isrForCrosstalkSources_999142_15.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 7789055d87315b276df243d72d208d85 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a518a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1a8f9228-d12f-4559-898c-9fb440a193d5_isrForCrosstalkSources_999142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1a8f9228-d12f-4559-898c-9fb440a193d5_isrForCrosstalkSources_999142_33.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash cc2211881cd62c56ef12af23ce47a861 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/71dc27e9-05a8-44c5-9ec0-63cc5be17a64_isrForCrosstalkSources_999143_30.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/71dc27e9-05a8-44c5-9ec0-63cc5be17a64_isrForCrosstalkSources_999143_30.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash f05057931b2db25e8ad45e865a3d0406 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/cc0a19b4-ff79-4398-abd7-abc6cd7b93b6_isrForCrosstalkSources_999142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/cc0a19b4-ff79-4398-abd7-abc6cd7b93b6_isrForCrosstalkSources_999142_17.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 935f1b8ddc360b21ffdcdda2ddbba0cc parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a86461bd-6a78-4bb5-a46e-b3c00091a0d6_isrForCrosstalkSources_999144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a86461bd-6a78-4bb5-a46e-b3c00091a0d6_isrForCrosstalkSources_999144_44.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 08de5e1633795066d73e563f775ac4b7 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20658dfe-61eb-4fea-b249-28aa3b061154_isrForCrosstalkSources_999144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20658dfe-61eb-4fea-b249-28aa3b061154_isrForCrosstalkSources_999144_12.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 742feb358cf5c93359231e0fa5d4eb76 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3c79ee17-5be1-4c1a-b036-35b358920fd1_isrForCrosstalkSources_999144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3c79ee17-5be1-4c1a-b036-35b358920fd1_isrForCrosstalkSources_999144_6.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 73fdcc70d8f2349da56e4e62dcf09d1f parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/914a11f8-d607-48bf-83f5-ae5325f64495_isrForCrosstalkSources_999144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/914a11f8-d607-48bf-83f5-ae5325f64495_isrForCrosstalkSources_999144_55.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash b6f2898cfa0e8f244004e1b5a6ffef50 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/c2770ab7-e27b-4408-a2f9-84942bfac642_isrForCrosstalkSources_999144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/c2770ab7-e27b-4408-a2f9-84942bfac642_isrForCrosstalkSources_999144_17.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 5987bdb0a88619b06bcdb0b064674ffa parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/82af21f0-b883-4579-af24-b5d72fa138ca_isrForCrosstalkSources_999144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/82af21f0-b883-4579-af24-b5d72fa138ca_isrForCrosstalkSources_999144_29.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 80bb9f65141f30284144d7f400288a51 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20770d34-d29d-44e9-8705-2ec102a44365_isrForCrosstalkSources_999144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20770d34-d29d-44e9-8705-2ec102a44365_isrForCrosstalkSources_999144_38.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash dbb59bf110ae2c415a0708a87ee7decf parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a51ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/03e86b34-b987-42b9-b83a-beee60df34ed_isrForCrosstalkSources_999144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/03e86b34-b987-42b9-b83a-beee60df34ed_isrForCrosstalkSources_999144_45.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 5e3514ff95e073e09e62242e94058005 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/df5eefc9-aded-4fab-bcaa-b24b255dad28_isrForCrosstalkSources_999142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/df5eefc9-aded-4fab-bcaa-b24b255dad28_isrForCrosstalkSources_999142_28.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 31a4456558531a57e9b9dabeada5d7f7 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a520c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b45411e2-5e7c-4c09-b224-60023987e844_isrForCrosstalkSources_999143_14.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b45411e2-5e7c-4c09-b224-60023987e844_isrForCrosstalkSources_999143_14.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 39ad49cae1811e81ec40c34d49c921a9 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/dfc2b17e-7076-4507-a61e-12202622ce4c_isrForCrosstalkSources_999142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/dfc2b17e-7076-4507-a61e-12202622ce4c_isrForCrosstalkSources_999142_25.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash da86c66ef475ad9e18b05771fc1d6b7a parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a522a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/04309b92-a5b6-421b-8522-c6ca73944bea_isrForCrosstalkSources_999143_13.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/04309b92-a5b6-421b-8522-c6ca73944bea_isrForCrosstalkSources_999143_13.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 49e370880a844642d717ad44a3ce3573 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4a88a7a6-dc2c-4e8d-b9c9-a5f5031a3847_isrForCrosstalkSources_999142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4a88a7a6-dc2c-4e8d-b9c9-a5f5031a3847_isrForCrosstalkSources_999142_36.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash f9716fad1565650109d06d94b495b788 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4e7c5106-2abe-49cf-9d05-0bf78dfac037_isrForCrosstalkSources_999144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4e7c5106-2abe-49cf-9d05-0bf78dfac037_isrForCrosstalkSources_999144_3.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 98722e87b6ff9b068ea959e3afc30b44 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a523e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77ee0de7-e749-4ced-88e7-307e829f2940_isrForCrosstalkSources_999144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77ee0de7-e749-4ced-88e7-307e829f2940_isrForCrosstalkSources_999144_1.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash 206062cd4e2568c403258a5218c031cf parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/298848ab-45f0-4775-b994-269b21e53114_isrForCrosstalkSources_999144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/298848ab-45f0-4775-b994-269b21e53114_isrForCrosstalkSources_999144_37.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 2ea1ebfe0a5244149849c72cac8bce98 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e9511e4b-f67c-4e9f-ba82-623a2c39a997_isrForCrosstalkSources_999143_29.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e9511e4b-f67c-4e9f-ba82-623a2c39a997_isrForCrosstalkSources_999143_29.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash b00ab0e79c1b0dc1c5bee4cb2fee6e9d parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3615f9e4-051c-4569-baed-c82a0e4d7d75_isrForCrosstalkSources_999144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3615f9e4-051c-4569-baed-c82a0e4d7d75_isrForCrosstalkSources_999144_24.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash dbe659d212de237f55ccd033799faceb parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/953ad453-92a3-48be-823b-17b885250ab7_isrForCrosstalkSources_999144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/953ad453-92a3-48be-823b-17b885250ab7_isrForCrosstalkSources_999144_43.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 18c77a386587202cf2b226523d5ccd87 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/845acd14-b5c2-466e-b147-39af6eec893e_isrForCrosstalkSources_999144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/845acd14-b5c2-466e-b147-39af6eec893e_isrForCrosstalkSources_999144_53.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 2359e42a296bbe45c0f9d9201379b9b9 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a527a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c92ec5b9-5b71-4bd9-abf0-b39ffa8b27d9_isrForCrosstalkSources_999143_50.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c92ec5b9-5b71-4bd9-abf0-b39ffa8b27d9_isrForCrosstalkSources_999143_50.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash ac279815c9213524cf069f7bbedd1d07 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a528e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/554370a1-66fb-427d-8f79-c2f031fc44d8_isrForCrosstalkSources_999144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/554370a1-66fb-427d-8f79-c2f031fc44d8_isrForCrosstalkSources_999144_48.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 9d0b1af7d90b988a96c0e68d27df3ff2 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44a947a9-46ad-4687-b1fe-d04e7c27b28d_isrForCrosstalkSources_999142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44a947a9-46ad-4687-b1fe-d04e7c27b28d_isrForCrosstalkSources_999142_45.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 13a488eb70329076e668c5d86a27a7cf parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/12b5982d-3d81-48b7-9501-eb0e8c3f911f_isrForCrosstalkSources_999142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/12b5982d-3d81-48b7-9501-eb0e8c3f911f_isrForCrosstalkSources_999142_31.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash bdc7df19e19b6e9ec21e31bac3450293 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/00658c3c-17f8-47aa-b8f3-9478a5a2c18e_isrForCrosstalkSources_999144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/00658c3c-17f8-47aa-b8f3-9478a5a2c18e_isrForCrosstalkSources_999144_32.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash bf173e7402d3a11fcb5436580a4b218f parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7eb08208-8793-4086-acbe-7860f85e5a91_isrForCrosstalkSources_999144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7eb08208-8793-4086-acbe-7860f85e5a91_isrForCrosstalkSources_999144_7.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash cefbaa161505d2dfeb77f951bc80c96e parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d5bc3116-0270-45a5-9bef-56fa4f04a984_isrForCrosstalkSources_999144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d5bc3116-0270-45a5-9bef-56fa4f04a984_isrForCrosstalkSources_999144_46.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 586344d8cfaa0b224ac06de3abe17db5 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/edefcdcd-caf8-4ba0-8f85-2adf26d87da7_isrForCrosstalkSources_999143_36.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/edefcdcd-caf8-4ba0-8f85-2adf26d87da7_isrForCrosstalkSources_999143_36.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash f1759e0b946bd488771b5b24f50f5d1f parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7e263e5b-efed-49b0-b389-26a25f2dbb47_isrForCrosstalkSources_999144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7e263e5b-efed-49b0-b389-26a25f2dbb47_isrForCrosstalkSources_999144_20.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 49d2868292e0ed72f36fffd1f55de283 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/cecceb18-0902-4471-9bbc-31b9b6d71d2c_isrForCrosstalkSources_999143_53.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/cecceb18-0902-4471-9bbc-31b9b6d71d2c_isrForCrosstalkSources_999143_53.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 2e707f3226cf89d752924d536ae78edb parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f81b5bb7-25f0-48b8-97cb-79c946440aef_isrForCrosstalkSources_999142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f81b5bb7-25f0-48b8-97cb-79c946440aef_isrForCrosstalkSources_999142_10.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 6bee880a9654d7cd2f03d9a8a25943be parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6b3575e0-f48f-4773-9d0c-bcaf0be2ad3b_isrForCrosstalkSources_999143_40.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6b3575e0-f48f-4773-9d0c-bcaf0be2ad3b_isrForCrosstalkSources_999143_40.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash e9c8b336c4c889e9cada537cf7d70844 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/eea033f8-3954-46dd-9e06-02ed65860dd2_isrForCrosstalkSources_999143_3.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/eea033f8-3954-46dd-9e06-02ed65860dd2_isrForCrosstalkSources_999143_3.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash d106f44199b34dcc0d523d1ccebb4cb1 parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a52fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/33f43d56-f44c-4b89-b4ac-69c8642412a4_isrForCrosstalkSources_999142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/33f43d56-f44c-4b89-b4ac-69c8642412a4_isrForCrosstalkSources_999142_16.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash fc5b62180412f73d350b2c3185c4d4f6 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ab7db3ed-7729-45da-b113-e22451bca9fe_isrForCrosstalkSources_999143_38.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ab7db3ed-7729-45da-b113-e22451bca9fe_isrForCrosstalkSources_999143_38.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 168476ffad48a8150ef42b69e57ad1a4 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1c9aefe1-3da2-427f-88e7-7a6369926efc_isrForCrosstalkSources_999144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1c9aefe1-3da2-427f-88e7-7a6369926efc_isrForCrosstalkSources_999144_23.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 0f037beb7ab1ce20e26c84493e641bcf parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1d5741b7-50a2-4bfa-9a71-c074e1f50127_isrForCrosstalkSources_999144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1d5741b7-50a2-4bfa-9a71-c074e1f50127_isrForCrosstalkSources_999144_9.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash e5203687a052de61d1f22c079163d5c4 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a531a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/26e53d73-69bd-47ec-93dc-0b9b77720b97_isrForCrosstalkSources_999142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/26e53d73-69bd-47ec-93dc-0b9b77720b97_isrForCrosstalkSources_999142_49.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 60fa5a4e90b16c4146c5165367646aac parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a532e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54605ca1-ab03-42e6-bf33-86ddfeae53be_isrForCrosstalkSources_999144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54605ca1-ab03-42e6-bf33-86ddfeae53be_isrForCrosstalkSources_999144_60.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash a70f39e5ebe582848cfa079ead44b05c parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bcaeabe-6b2b-4563-b6fa-d2ec87c02fd7_isrForCrosstalkSources_999142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bcaeabe-6b2b-4563-b6fa-d2ec87c02fd7_isrForCrosstalkSources_999142_50.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 72be19bba8507bcad15a359a0eefc7bb parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/30757288-0d8c-49ae-b4cb-5fe4624bf38e_isrForCrosstalkSources_999144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/30757288-0d8c-49ae-b4cb-5fe4624bf38e_isrForCrosstalkSources_999144_16.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash aaf2de9e86d14eca4d70c98343f29b06 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8bd22da7-6936-4f45-8866-5fae596598c5_isrForCrosstalkSources_999142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8bd22da7-6936-4f45-8866-5fae596598c5_isrForCrosstalkSources_999142_30.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 9be13aed23bb9895cb99d1078dd763c4 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ee65a551-30d0-4400-b92e-d6d58d218ad6_isrForCrosstalkSources_999143_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ee65a551-30d0-4400-b92e-d6d58d218ad6_isrForCrosstalkSources_999143_19.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 9dfeb5638d64dfc7b4a8993bdcdbcea7 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fddcfb40-01a2-4a11-804a-e7995684f24c_isrForCrosstalkSources_999142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fddcfb40-01a2-4a11-804a-e7995684f24c_isrForCrosstalkSources_999142_43.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 35e2c494d82cea3eef2f3475ef7ce473 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a536a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/24ed63bd-cdf8-4760-8843-c03526fa220c_isrForCrosstalkSources_999143_46.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/24ed63bd-cdf8-4760-8843-c03526fa220c_isrForCrosstalkSources_999143_46.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash fa0573153f7646040173c69e56299ff5 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4cfdea54-ac79-4208-881d-2b541502b4c3_isrForCrosstalkSources_999142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4cfdea54-ac79-4208-881d-2b541502b4c3_isrForCrosstalkSources_999142_35.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 47eda74424f0c21a1b0080fb8295f0e4 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/80ae7cc8-84ea-4d20-bd60-f57a64f595f8_isrForCrosstalkSources_999143_1.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/80ae7cc8-84ea-4d20-bd60-f57a64f595f8_isrForCrosstalkSources_999143_1.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash c155c0229f879260d9ca309a89217794 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a537e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4d7e609d-8e27-4d78-9c6a-24bd886cb746_isrForCrosstalkSources_999144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4d7e609d-8e27-4d78-9c6a-24bd886cb746_isrForCrosstalkSources_999144_33.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash dfef5d70f0c51b61df9b34e43f1bf549 parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54ff4baf-d2d1-40f4-a01d-6e89d8996b68_isrForCrosstalkSources_999144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54ff4baf-d2d1-40f4-a01d-6e89d8996b68_isrForCrosstalkSources_999144_52.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash c32659cbf2e8720425752031d776360a parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a539c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a5f902d5-157a-409d-b7ea-3138dabb2c0c_isrForCrosstalkSources_999144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a5f902d5-157a-409d-b7ea-3138dabb2c0c_isrForCrosstalkSources_999144_39.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash a96dfd4390584b7075d0a06609e9dcaa parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5a5150c1-9640-459e-b3d5-05520279001e_isrForCrosstalkSources_999143_35.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5a5150c1-9640-459e-b3d5-05520279001e_isrForCrosstalkSources_999143_35.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash c05aacefa875ec9250803facf9f89d31 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f3f2a3f9-e3eb-491c-ac62-210bf6de780b_isrForCrosstalkSources_999144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f3f2a3f9-e3eb-491c-ac62-210bf6de780b_isrForCrosstalkSources_999144_31.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 96055f2894198132f40dedb338b7a6ae parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d604db69-14dd-4376-8d21-c3d71430e94b_isrForCrosstalkSources_999142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d604db69-14dd-4376-8d21-c3d71430e94b_isrForCrosstalkSources_999142_38.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash adb2b6a7f47006d65bf2c533b461cc1a parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4afdd38f-b874-4252-9383-317feda0c28f_isrForCrosstalkSources_999143_16.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4afdd38f-b874-4252-9383-317feda0c28f_isrForCrosstalkSources_999143_16.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash acd75d90f041900098a6ed4c80e19929 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b968394c-4330-464e-a551-67707783f694_isrForCrosstalkSources_999143_60.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b968394c-4330-464e-a551-67707783f694_isrForCrosstalkSources_999143_60.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 2f6933e7b7fc2a835a9e3473244cfd5e parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/253ca2b5-89e7-4c07-a566-eae92bb29cf2_isrForCrosstalkSources_999144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/253ca2b5-89e7-4c07-a566-eae92bb29cf2_isrForCrosstalkSources_999144_19.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 0e3e59df1871f24ab8211741ebab83bf parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3050de73-8be0-4b23-9ff7-cd1a25468413_isrForCrosstalkSources_999143_45.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3050de73-8be0-4b23-9ff7-cd1a25468413_isrForCrosstalkSources_999143_45.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 2a09927189cfdd5b639f94dfd6ad26f9 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0ca6fbc1-8019-4600-b3c5-55821708c334_isrForCrosstalkSources_999142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0ca6fbc1-8019-4600-b3c5-55821708c334_isrForCrosstalkSources_999142_60.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash bb4d3a1ecd13a5700133b466879ce79c parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a50680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5fae907e-2c3c-4638-bd48-181d407cf95f_isrForCrosstalkSources_999143_8.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5fae907e-2c3c-4638-bd48-181d407cf95f_isrForCrosstalkSources_999143_8.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash e1c340cd985f23a11ed99e099ff2492a parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1e18a3-3a89-455c-9a15-a3e0422a5653_isrForCrosstalkSources_999142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1e18a3-3a89-455c-9a15-a3e0422a5653_isrForCrosstalkSources_999142_12.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 33dcbb7d31bc6577d030595803a9dec3 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9570c945-ae54-48e2-93ae-3d0f0865ca0a_isrForCrosstalkSources_999142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9570c945-ae54-48e2-93ae-3d0f0865ca0a_isrForCrosstalkSources_999142_32.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 70e49565d9d4f0388e9dd20ca0f24ed7 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e06fa877-50d7-4d29-82e1-2dff75ed09c2_isrForCrosstalkSources_999144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e06fa877-50d7-4d29-82e1-2dff75ed09c2_isrForCrosstalkSources_999144_40.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 18615298705f706109ccd6d7d5d45eb2 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dedce3e-1b09-4027-b116-4dc544e3a9f2_isrForCrosstalkSources_999142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dedce3e-1b09-4027-b116-4dc544e3a9f2_isrForCrosstalkSources_999142_56.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash af05d20e6edc7531ab513bae0a735db4 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/020c3b8a-7140-49d8-9019-cfac0065f683_isrForCrosstalkSources_999142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/020c3b8a-7140-49d8-9019-cfac0065f683_isrForCrosstalkSources_999142_2.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 2fd3f4b0c5a2b49616857857640a92c3 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dc4e906-1837-4aa8-87b5-fad696e5d9ef_isrForCrosstalkSources_999142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dc4e906-1837-4aa8-87b5-fad696e5d9ef_isrForCrosstalkSources_999142_6.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 9eaf6a7c29615b7c7fc57e4323744469 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b2067b94-7b05-4fa7-ab48-093d3ded6fe1_isrForCrosstalkSources_999142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b2067b94-7b05-4fa7-ab48-093d3ded6fe1_isrForCrosstalkSources_999142_24.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash befbd47659390b6e07250d0430d93abf parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ac25ebff-24f8-4938-ad07-89a16a1d1d5c_isrForCrosstalkSources_999143_42.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ac25ebff-24f8-4938-ad07-89a16a1d1d5c_isrForCrosstalkSources_999143_42.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 03f1d0561e5efe474724f57e4911f595 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9ad3772c-4e36-49e5-82f6-e9a16ad3800f_isrForCrosstalkSources_999143_20.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9ad3772c-4e36-49e5-82f6-e9a16ad3800f_isrForCrosstalkSources_999143_20.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 9937eeb7fcd2cb2cd57faad8fc817d49 parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f5882ca2-7ecd-4091-a235-bf755b78b69a_isrForCrosstalkSources_999143_37.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f5882ca2-7ecd-4091-a235-bf755b78b69a_isrForCrosstalkSources_999143_37.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash f328ed48691535ba4585a348a0770715 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8606a537-0712-441d-aa47-31c427fcea5a_isrForCrosstalkSources_999144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8606a537-0712-441d-aa47-31c427fcea5a_isrForCrosstalkSources_999144_2.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash ce580030da6870a2cc8aa3994ca727bb parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9c1ced00-0723-48bb-b041-772cfa004566_isrForCrosstalkSources_999142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9c1ced00-0723-48bb-b041-772cfa004566_isrForCrosstalkSources_999142_41.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash 752ee98fcf27df5b108215a93b9a8a44 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d171a9a6-282e-426a-8e60-2eeb30b765e3_isrForCrosstalkSources_999142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d171a9a6-282e-426a-8e60-2eeb30b765e3_isrForCrosstalkSources_999142_21.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash ae577de97ac2904582757a66fba9de2e parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/94e3f774-a75e-4398-bcf3-c5ec17d27681_isrForCrosstalkSources_999144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/94e3f774-a75e-4398-bcf3-c5ec17d27681_isrForCrosstalkSources_999144_10.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 3ea107bca0e38d3d2a10d27628763928 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5e7186c-e7fb-433c-9a1a-2bb2ff1af54b_isrForCrosstalkSources_999142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5e7186c-e7fb-433c-9a1a-2bb2ff1af54b_isrForCrosstalkSources_999142_61.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash a796d89276ce7075625fa06b78832411 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f43c0444-5e2a-47ad-86a1-fb4dec1e31c6_isrForCrosstalkSources_999143_54.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f43c0444-5e2a-47ad-86a1-fb4dec1e31c6_isrForCrosstalkSources_999143_54.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash d32c89cdab19aa053f5f51acced1d731 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/924c30cb-cd12-4bf0-bfaf-59b03cf1b9c7_isrForCrosstalkSources_999143_23.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/924c30cb-cd12-4bf0-bfaf-59b03cf1b9c7_isrForCrosstalkSources_999143_23.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 19157c463d680ab86babbbe2280a1e10 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/87e1d534-7446-400f-a1d9-7d3bebf86ed4_isrForCrosstalkSources_999143_43.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/87e1d534-7446-400f-a1d9-7d3bebf86ed4_isrForCrosstalkSources_999143_43.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash d7bf17ee0ac2113d047fbf4997086e1e parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/adea87f4-e295-4c51-8693-9e1e550e72c3_isrForCrosstalkSources_999143_59.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/adea87f4-e295-4c51-8693-9e1e550e72c3_isrForCrosstalkSources_999143_59.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 1ca5402a3ea2fafca97edc6fd243997e parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f402e564-10b5-4b6a-a01f-27a8eab1e3ee_isrForCrosstalkSources_999142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f402e564-10b5-4b6a-a01f-27a8eab1e3ee_isrForCrosstalkSources_999142_42.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash bdaff7547ff21fd19904367a134f61f1 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e6e6b34a-655d-4863-a7aa-a9e4ee3cb230_isrForCrosstalkSources_999144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e6e6b34a-655d-4863-a7aa-a9e4ee3cb230_isrForCrosstalkSources_999144_59.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 4643718452293de5321e63fa391a0975 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0f3b12dc-cea3-4e65-a3c6-481b194d9c57_isrForCrosstalkSources_999143_32.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0f3b12dc-cea3-4e65-a3c6-481b194d9c57_isrForCrosstalkSources_999143_32.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 5ad0c24b4d1a2d6e1aec9caed3057773 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0cb31b1b-18c5-4f2c-8302-1ce59eaeee20_isrForCrosstalkSources_999143_33.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0cb31b1b-18c5-4f2c-8302-1ce59eaeee20_isrForCrosstalkSources_999143_33.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 6026762f4e5853004d6828f542cd08d7 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/492887fd-3c8e-4a72-946c-47c3757725e5_isrForCrosstalkSources_999142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/492887fd-3c8e-4a72-946c-47c3757725e5_isrForCrosstalkSources_999142_62.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 023904a7ee9e32318a99aa6f83cdc889 parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f2ff2acc-eb8d-454b-b982-837cb63c3af3_isrForCrosstalkSources_999144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f2ff2acc-eb8d-454b-b982-837cb63c3af3_isrForCrosstalkSources_999144_13.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 315839b7f9f8f810ecc386d3a96a0f2e parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/746ac36c-106e-4ebf-b6cf-8e59e9897b77_isrForCrosstalkSources_999143_51.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/746ac36c-106e-4ebf-b6cf-8e59e9897b77_isrForCrosstalkSources_999143_51.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 1fafadf507a92249fd88c5edde5a54f2 parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c5c18299-2ee2-4e0e-a47c-3f64f560e4eb_isrForCrosstalkSources_999142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c5c18299-2ee2-4e0e-a47c-3f64f560e4eb_isrForCrosstalkSources_999142_58.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash f8ac4ad2a638e9ecca9bbaaf0002f397 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02edaf65-ac25-4324-8a6b-2163fca5ce6c_isrForCrosstalkSources_999142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02edaf65-ac25-4324-8a6b-2163fca5ce6c_isrForCrosstalkSources_999142_1.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash ef18854f3776276924746b82caf5bc17 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8231a74d-6eaf-4112-ac5a-d7442daed0e1_isrForCrosstalkSources_999142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8231a74d-6eaf-4112-ac5a-d7442daed0e1_isrForCrosstalkSources_999142_46.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 32a4397f74a0c619fbaefc965ff0ffe8 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c8b36c90-c604-4503-93f5-bbceeb72e348_isrForCrosstalkSources_999142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c8b36c90-c604-4503-93f5-bbceeb72e348_isrForCrosstalkSources_999142_14.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash e5f323ed7150bb6cadac63feb2167101 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/bc22dd85-a85e-40d5-90b6-a0431671b0bd_isrForCrosstalkSources_999143_5.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/bc22dd85-a85e-40d5-90b6-a0431671b0bd_isrForCrosstalkSources_999143_5.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 23a1207d719d651e0ed90a8ba5e27350 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/082a6cfa-d11e-46fb-9fdb-66af283667b9_isrForCrosstalkSources_999143_55.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/082a6cfa-d11e-46fb-9fdb-66af283667b9_isrForCrosstalkSources_999143_55.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash e2b9c59f6fd3eaf93913fcb5cff768dd parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5925ff58-1736-45c7-889a-538ef4a8e548_isrForCrosstalkSources_999144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5925ff58-1736-45c7-889a-538ef4a8e548_isrForCrosstalkSources_999144_22.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 8f551d58964722b60ebe00fa5090515c parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/aaaea80d-0a1e-4873-94e2-edd7c64795d2_isrForCrosstalkSources_999142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/aaaea80d-0a1e-4873-94e2-edd7c64795d2_isrForCrosstalkSources_999142_48.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 4e95f51d666c1375a7db85f9709f8689 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f46fb3c9-ae8a-4248-9420-036e2d44411d_isrForCrosstalkSources_999144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f46fb3c9-ae8a-4248-9420-036e2d44411d_isrForCrosstalkSources_999144_42.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 3ee97c77c538897fb426c35615eeb0a5 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-16 00:06:00 proc_lsst.multi:146 [INFO] found job 17843 in provider local proc_lsst.multi INFO: found job 17843 in provider local parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d1bba610-6846-4065-9192-40c222112640_isrForCrosstalkSources_999143_44.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d1bba610-6846-4065-9192-40c222112640_isrForCrosstalkSources_999143_44.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 571244be64661ed5aa54d3f6f083a121 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0a3a4145-2196-4954-8040-084b309660fb_isrForCrosstalkSources_999142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0a3a4145-2196-4954-8040-084b309660fb_isrForCrosstalkSources_999142_55.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 02fa79fe59e872c8475b46320e0e5890 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c6e55db6-e30f-48dc-a1ae-46c8e294bc8d_isrForCrosstalkSources_999143_48.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c6e55db6-e30f-48dc-a1ae-46c8e294bc8d_isrForCrosstalkSources_999143_48.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash cac4cabfa24a648de629fd328650af0a parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77468676-0b7d-4fbb-8b97-33ae2fb0d1ac_isrForCrosstalkSources_999144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77468676-0b7d-4fbb-8b97-33ae2fb0d1ac_isrForCrosstalkSources_999144_57.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 89c52e80b7d20fbdea26f0c4285bb5a7 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4dd0d19a-a10d-4383-bf9b-4c74d0785778_isrForCrosstalkSources_999143_7.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4dd0d19a-a10d-4383-bf9b-4c74d0785778_isrForCrosstalkSources_999143_7.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 2e547829d90558c014827f603f2ae862 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/28e458fd-c8f6-4862-8331-a29e4a9e168f_isrForCrosstalkSources_999144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/28e458fd-c8f6-4862-8331-a29e4a9e168f_isrForCrosstalkSources_999144_27.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 48a579dd3f253e7a7fc36a423d00a209 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1ea51f-95b4-4fd2-b20a-ce0a12463d14_isrForCrosstalkSources_999142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1ea51f-95b4-4fd2-b20a-ce0a12463d14_isrForCrosstalkSources_999142_19.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash d2a0e4d8c8b43f6309065ce0f09b606b parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0a6cc0a7-2fad-4c14-b536-26fbb205d50b_isrForCrosstalkSources_999144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0a6cc0a7-2fad-4c14-b536-26fbb205d50b_isrForCrosstalkSources_999144_41.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash 59cef603aeb340ea6ad3a25e84929f4e parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0b2ba3c-13c8-4c17-8bfa-6ae70ab22ed4_isrForCrosstalkSources_999142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0b2ba3c-13c8-4c17-8bfa-6ae70ab22ed4_isrForCrosstalkSources_999142_3.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 14fb16b742bc93f3e689cbd2a34f6729 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fc8f47df-a3c7-4829-9b86-69eaf973702c_isrForCrosstalkSources_999142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fc8f47df-a3c7-4829-9b86-69eaf973702c_isrForCrosstalkSources_999142_57.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 7e5b14d4be788d6d8e5a941df73ad7ee parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/7591b490-6b59-4c93-9cdb-69eca0639aff_isrForCrosstalkSources_999142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/7591b490-6b59-4c93-9cdb-69eca0639aff_isrForCrosstalkSources_999142_26.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 5d9bac80be32b0f7834c2bc25617b773 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/502e980f-a2e5-4c35-8d56-73164369effb_isrForCrosstalkSources_999144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/502e980f-a2e5-4c35-8d56-73164369effb_isrForCrosstalkSources_999144_18.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 723a23a5af73cee7499d9044a9f61ca8 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8444bbc6-61fa-4a18-a537-d3f17844c0d3_isrForCrosstalkSources_999144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8444bbc6-61fa-4a18-a537-d3f17844c0d3_isrForCrosstalkSources_999144_61.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 12c6df995d0947a7f3823ca82a68d373 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3191c074-d861-42e2-8bf7-52eb19223aa0_isrForCrosstalkSources_999144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3191c074-d861-42e2-8bf7-52eb19223aa0_isrForCrosstalkSources_999144_4.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 43bf6d59c105343a2ecee985176cd24b parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8fdeb541-3d8c-48fd-96ad-d314bad773b1_isrForCrosstalkSources_999143_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8fdeb541-3d8c-48fd-96ad-d314bad773b1_isrForCrosstalkSources_999143_61.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash f7983d496253f514223197d72349f696 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/edb2997e-aba5-45e7-afdf-eab98be04049_isrForCrosstalkSources_999144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/edb2997e-aba5-45e7-afdf-eab98be04049_isrForCrosstalkSources_999144_25.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 82745aa3bffcc8991c18a0ac2373e7a8 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f524fd6a-1235-4aaf-8b10-2d8eed6bfa0a_isrForCrosstalkSources_999143_12.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f524fd6a-1235-4aaf-8b10-2d8eed6bfa0a_isrForCrosstalkSources_999143_12.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 2f0f2780d323e19614d8d5e26aba0c4b parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/857c0452-e5b3-4142-b164-451382bfcb90_isrForCrosstalkSources_999143_24.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/857c0452-e5b3-4142-b164-451382bfcb90_isrForCrosstalkSources_999143_24.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 4c3211fea43f702dac3a4ffd48624653 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d4ab1c1a-bd62-44df-ad22-50b4a8931a95_isrForCrosstalkSources_999144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d4ab1c1a-bd62-44df-ad22-50b4a8931a95_isrForCrosstalkSources_999144_26.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash 941a9abec23446c929d7fe3a278ca34e parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 161 parsl.dataflow.strategy DEBUG: Executor multi has 161 active tasks, 1/0 running/pending blocks, and 0 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 40 more blocks parsl.executors.status_handling INFO: Scaling out by 40 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-16 00:06:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-16 00:06:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:06:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/102bcc3e-b0cb-49ee-bbbc-7022b29f3a52_isrForCrosstalkSources_999143_22.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/102bcc3e-b0cb-49ee-bbbc-7022b29f3a52_isrForCrosstalkSources_999143_22.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 65d163f03adc936bcbd8f3395b378732 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/ecca6f78-2e44-4b03-bc08-24e76ed425a0_isrForCrosstalkSources_999144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/ecca6f78-2e44-4b03-bc08-24e76ed425a0_isrForCrosstalkSources_999144_47.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 5fe4d130720c05f78df2c105893c6905 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4a842dba-3fb6-4910-be56-9f66abe93336_isrForCrosstalkSources_999143_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4a842dba-3fb6-4910-be56-9f66abe93336_isrForCrosstalkSources_999143_52.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash c43e004c18eda71db4c1e31e89eef5d8 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c8f4070a-76ae-4594-ad7c-52e0e2fadef8_isrForCrosstalkSources_999143_18.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c8f4070a-76ae-4594-ad7c-52e0e2fadef8_isrForCrosstalkSources_999143_18.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 5437ebd40444605aa246032e6e0c6556 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe81a4e-7572-4ff7-9d42-fd12d378d7ee_isrForCrosstalkSources_999143_58.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe81a4e-7572-4ff7-9d42-fd12d378d7ee_isrForCrosstalkSources_999143_58.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 7fe79a1addfd058cff455ba3e948ed7d parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52a53f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9a777d70-0b16-4cd9-bc57-64bb85496ce0_isrForCrosstalkSources_999143_4.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9a777d70-0b16-4cd9-bc57-64bb85496ce0_isrForCrosstalkSources_999143_4.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash eff8f558c77de51b4a6bc472c7e3dcf6 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f51511c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d6aaab65-34aa-4594-beb7-67637e2e7af9_isrForCrosstalkSources_999144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d6aaab65-34aa-4594-beb7-67637e2e7af9_isrForCrosstalkSources_999144_56.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash c12c426ac974efa13f1bcb17a5be91a3 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/11409320-efd1-4bf5-995b-67995e5b2026_isrForCrosstalkSources_999142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/11409320-efd1-4bf5-995b-67995e5b2026_isrForCrosstalkSources_999142_4.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash e3d537754504a8c7585f8a7b1efe32e4 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f51511da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/1c9224ac-a1ef-4e42-b4ef-705ab8b063bd_isrForCrosstalkSources_999143_25.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/1c9224ac-a1ef-4e42-b4ef-705ab8b063bd_isrForCrosstalkSources_999143_25.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash b3a5c7ef824f724e5848693dc706773b parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f51511bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44695da1-ed6d-4b87-a7af-1e488f4d3d8e_isrForCrosstalkSources_999142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44695da1-ed6d-4b87-a7af-1e488f4d3d8e_isrForCrosstalkSources_999142_37.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash 6872534cb32fdf52d450ba343f47ff69 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f51511b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6f19dae7-746b-42de-a9b0-2905c1749ad0_isrForCrosstalkSources_999143_34.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6f19dae7-746b-42de-a9b0-2905c1749ad0_isrForCrosstalkSources_999143_34.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash bc04e03e0351277d9ec617a5033ee338 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e4018291-1c2e-4090-b099-6f8e858d1f27_isrForCrosstalkSources_999143_47.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e4018291-1c2e-4090-b099-6f8e858d1f27_isrForCrosstalkSources_999143_47.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 6714bee537022d96a7ac05e094ade36c parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/447720a1-c4de-425b-bf38-d29a2b9ec0f7_isrForCrosstalkSources_999142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/447720a1-c4de-425b-bf38-d29a2b9ec0f7_isrForCrosstalkSources_999142_8.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 1832cced40e5b53c76e8b5d3c75ed668 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/00f8f14d-b211-48d7-b7a8-a4474df17d29_isrForCrosstalkSources_999142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/00f8f14d-b211-48d7-b7a8-a4474df17d29_isrForCrosstalkSources_999142_23.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 540bc302d06b22cb867a816cfecfd158 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/07c84200-7544-4302-8538-c4e47dfe29d1_isrForCrosstalkSources_999144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/07c84200-7544-4302-8538-c4e47dfe29d1_isrForCrosstalkSources_999144_36.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash ad0f3db5506e77b5f09b7863526117fa parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0ffd96e6-401f-421d-befe-cf49108dacf5_isrForCrosstalkSources_999144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0ffd96e6-401f-421d-befe-cf49108dacf5_isrForCrosstalkSources_999144_35.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash b09db34f3e44a39de625cb73d63b6ea2 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/efd21f7f-c233-4289-b29e-244de8b6d35c_isrForCrosstalkSources_999144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/efd21f7f-c233-4289-b29e-244de8b6d35c_isrForCrosstalkSources_999144_50.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 75b7b2271bacb19582a80777227cf42d parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b439e7a8-db2b-49ec-8d5d-9172cb96de5c_isrForCrosstalkSources_999143_62.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b439e7a8-db2b-49ec-8d5d-9172cb96de5c_isrForCrosstalkSources_999143_62.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 7eead6fa25e8bd8eaee99d4eac4d2ec7 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/91950290-447b-4f31-98d8-df74e53ef041_isrForCrosstalkSources_999144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/91950290-447b-4f31-98d8-df74e53ef041_isrForCrosstalkSources_999144_34.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 68b486cccc2228420e8eb948f1aa1843 parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e432a7d4-1762-4992-9237-53708d143125_isrForCrosstalkSources_999143_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e432a7d4-1762-4992-9237-53708d143125_isrForCrosstalkSources_999143_27.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash baef1cd603584cf44e48720a2aa2dfc2 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/73b32838-bba0-4736-8286-19ce3bbb9a25_isrForCrosstalkSources_999142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/73b32838-bba0-4736-8286-19ce3bbb9a25_isrForCrosstalkSources_999142_39.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash b6b56e77ae7e128aa2cd7cdffeea25ef parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bea4b9a-0421-4f7c-b13e-188b834699fb_isrForCrosstalkSources_999142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bea4b9a-0421-4f7c-b13e-188b834699fb_isrForCrosstalkSources_999142_9.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash bc691a3a98a01a46e86240adf4fd8276 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x147f52ad2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/df9aa7be-903f-4ef6-83d6-5f98379fbe0d_isrForCrosstalkSources_999143_17.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/df9aa7be-903f-4ef6-83d6-5f98379fbe0d_isrForCrosstalkSources_999143_17.stderr 2024-03-16 00:06:02 proc_lsst.multi:170 [INFO] [multi] job_id 17096750 proc_lsst.multi INFO: [multi] job_id 17096750 2024-03-16 00:06:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096750 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096750 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17096750 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-16 00:06:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:02 proc_lsst.multi:170 [INFO] [multi] job_id 17096751 proc_lsst.multi INFO: [multi] job_id 17096751 2024-03-16 00:06:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096751 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096751 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17096751 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-16 00:06:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:02 proc_lsst.multi:170 [INFO] [multi] job_id 17096752 proc_lsst.multi INFO: [multi] job_id 17096752 2024-03-16 00:06:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096752 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096752 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17096752 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-16 00:06:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:03 proc_lsst.multi:170 [INFO] [multi] job_id 17096753 proc_lsst.multi INFO: [multi] job_id 17096753 2024-03-16 00:06:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096753 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096753 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17096753 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-16 00:06:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-16 00:06:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:03 proc_lsst.multi:170 [INFO] [multi] job_id 17096754 proc_lsst.multi INFO: [multi] job_id 17096754 2024-03-16 00:06:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:03 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096754 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096754 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17096754 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-16 00:06:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-16 00:06:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:04 proc_lsst.multi:170 [INFO] [multi] job_id 17096755 proc_lsst.multi INFO: [multi] job_id 17096755 2024-03-16 00:06:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:06:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096755 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096755 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17096755 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-16 00:06:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:04 proc_lsst.multi:170 [INFO] [multi] job_id 17096756 proc_lsst.multi INFO: [multi] job_id 17096756 2024-03-16 00:06:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:06:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096756 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096756 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17096756 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-16 00:06:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:04 proc_lsst.multi:170 [INFO] [multi] job_id 17096757 proc_lsst.multi INFO: [multi] job_id 17096757 2024-03-16 00:06:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:06:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096757 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096757 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17096757 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-16 00:06:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:06:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:05 proc_lsst.multi:170 [INFO] [multi] job_id 17096758 proc_lsst.multi INFO: [multi] job_id 17096758 2024-03-16 00:06:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:06:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096758 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096758 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17096758 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-16 00:06:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-16 00:06:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:05 proc_lsst.multi:170 [INFO] [multi] job_id 17096759 proc_lsst.multi INFO: [multi] job_id 17096759 2024-03-16 00:06:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:06:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:05 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096759 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096759 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17096759 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-16 00:06:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-16 00:06:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:06:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:06 proc_lsst.multi:170 [INFO] [multi] job_id 17096760 proc_lsst.multi INFO: [multi] job_id 17096760 2024-03-16 00:06:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 00:06:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096760 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096760 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17096760 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-16 00:06:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:06 proc_lsst.multi:170 [INFO] [multi] job_id 17096761 proc_lsst.multi INFO: [multi] job_id 17096761 2024-03-16 00:06:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 00:06:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096761 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096761 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17096761 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-16 00:06:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:06 proc_lsst.multi:170 [INFO] [multi] job_id 17096762 proc_lsst.multi INFO: [multi] job_id 17096762 2024-03-16 00:06:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 00:06:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096762 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096762 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17096762 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-16 00:06:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-16 00:06:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:06 proc_lsst.multi:170 [INFO] [multi] job_id 17096763 proc_lsst.multi INFO: [multi] job_id 17096763 2024-03-16 00:06:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 00:06:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:06 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096763 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096763 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17096763 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-16 00:06:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-16 00:06:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:07 proc_lsst.multi:170 [INFO] [multi] job_id 17096764 proc_lsst.multi INFO: [multi] job_id 17096764 2024-03-16 00:06:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 00:06:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096764 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096764 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17096764 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-16 00:06:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:07 proc_lsst.multi:170 [INFO] [multi] job_id 17096765 proc_lsst.multi INFO: [multi] job_id 17096765 2024-03-16 00:06:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 00:06:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096765 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096765 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17096765 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-16 00:06:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:07 proc_lsst.multi:170 [INFO] [multi] job_id 17096766 proc_lsst.multi INFO: [multi] job_id 17096766 2024-03-16 00:06:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 00:06:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096766 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096766 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17096766 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-16 00:06:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-16 00:06:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:08 proc_lsst.multi:170 [INFO] [multi] job_id 17096767 proc_lsst.multi INFO: [multi] job_id 17096767 2024-03-16 00:06:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 00:06:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096767 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096767 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17096767 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-16 00:06:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-16 00:06:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-16 00:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:08 proc_lsst.multi:170 [INFO] [multi] job_id 17096768 proc_lsst.multi INFO: [multi] job_id 17096768 2024-03-16 00:06:08 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 00:06:08 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:08 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096768 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096768 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17096768 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-16 00:06:08 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-16 00:06:08 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:08 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:08 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-16 00:06:08 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:08 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:09 proc_lsst.multi:170 [INFO] [multi] job_id 17096769 proc_lsst.multi INFO: [multi] job_id 17096769 2024-03-16 00:06:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 00:06:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096769 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096769 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17096769 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-16 00:06:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:09 proc_lsst.multi:170 [INFO] [multi] job_id 17096770 proc_lsst.multi INFO: [multi] job_id 17096770 2024-03-16 00:06:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 00:06:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096770 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096770 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17096770 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-16 00:06:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:09 proc_lsst.multi:170 [INFO] [multi] job_id 17096771 proc_lsst.multi INFO: [multi] job_id 17096771 2024-03-16 00:06:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 00:06:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096771 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096771 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17096771 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-16 00:06:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-16 00:06:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:10 proc_lsst.multi:170 [INFO] [multi] job_id 17096772 proc_lsst.multi INFO: [multi] job_id 17096772 2024-03-16 00:06:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 00:06:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096772 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096772 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17096772 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-16 00:06:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:10 proc_lsst.multi:170 [INFO] [multi] job_id 17096773 proc_lsst.multi INFO: [multi] job_id 17096773 2024-03-16 00:06:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 00:06:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096773 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096773 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17096773 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-16 00:06:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:10 proc_lsst.multi:170 [INFO] [multi] job_id 17096774 proc_lsst.multi INFO: [multi] job_id 17096774 2024-03-16 00:06:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 00:06:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096774 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096774 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17096774 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-16 00:06:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-16 00:06:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6c638398b6054333c01112778d47aae with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/cc2fa00f-fe47-48a3-bf5b-a6eacb7614b0_isrForCrosstalkSources_999144_15.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/cc2fa00f-fe47-48a3-bf5b-a6eacb7614b0_isrForCrosstalkSources_999144_15.stderr 2024-03-16 00:06:11 proc_lsst.multi:170 [INFO] [multi] job_id 17096775 proc_lsst.multi INFO: [multi] job_id 17096775 2024-03-16 00:06:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 00:06:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096775 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096775 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17096775 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-16 00:06:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9e5c9b49077a673e6f4730809a0f6cc with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/6609c52d-6e7d-4e91-8838-0a0f759c7b69_isrForCrosstalkSources_999142_40.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/6609c52d-6e7d-4e91-8838-0a0f759c7b69_isrForCrosstalkSources_999142_40.stderr 2024-03-16 00:06:11 proc_lsst.multi:170 [INFO] [multi] job_id 17096776 proc_lsst.multi INFO: [multi] job_id 17096776 2024-03-16 00:06:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 00:06:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096776 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096776 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17096776 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-16 00:06:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:11 proc_lsst.multi:170 [INFO] [multi] job_id 17096777 proc_lsst.multi INFO: [multi] job_id 17096777 2024-03-16 00:06:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 00:06:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096777 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096777 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17096777 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-16 00:06:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-16 00:06:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:12 proc_lsst.multi:170 [INFO] [multi] job_id 17096778 proc_lsst.multi INFO: [multi] job_id 17096778 2024-03-16 00:06:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 00:06:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096778 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096778 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17096778 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-16 00:06:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-16 00:06:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-16 00:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:12 proc_lsst.multi:170 [INFO] [multi] job_id 17096779 proc_lsst.multi INFO: [multi] job_id 17096779 2024-03-16 00:06:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:12 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17096779 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17096779 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17096779 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-16 00:06:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-16 00:06:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:06:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:13 proc_lsst.multi:170 [INFO] [multi] job_id 17096780 proc_lsst.multi INFO: [multi] job_id 17096780 2024-03-16 00:06:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096780 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096780 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17096780 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-16 00:06:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:13 proc_lsst.multi:170 [INFO] [multi] job_id 17096781 proc_lsst.multi INFO: [multi] job_id 17096781 2024-03-16 00:06:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096781 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096781 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17096781 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-16 00:06:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:13 proc_lsst.multi:170 [INFO] [multi] job_id 17096782 proc_lsst.multi INFO: [multi] job_id 17096782 2024-03-16 00:06:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096782 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096782 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17096782 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-16 00:06:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:15 proc_lsst.multi:170 [INFO] [multi] job_id 17096787 proc_lsst.multi INFO: [multi] job_id 17096787 2024-03-16 00:06:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:15 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096787 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096787 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17096787 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-16 00:06:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:15 proc_lsst.multi:170 [INFO] [multi] job_id 17096788 proc_lsst.multi INFO: [multi] job_id 17096788 2024-03-16 00:06:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:15 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17096788 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17096788 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17096788 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-16 00:06:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-16 00:06:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:15 proc_lsst.multi:170 [INFO] [multi] job_id 17096789 proc_lsst.multi INFO: [multi] job_id 17096789 2024-03-16 00:06:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096789 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096789 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17096789 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-16 00:06:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:16 proc_lsst.multi:170 [INFO] [multi] job_id 17096790 proc_lsst.multi INFO: [multi] job_id 17096790 2024-03-16 00:06:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096790 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096790 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17096790 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-16 00:06:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:16 proc_lsst.multi:170 [INFO] [multi] job_id 17096791 proc_lsst.multi INFO: [multi] job_id 17096791 2024-03-16 00:06:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096791 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096791 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17096791 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-16 00:06:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:16 proc_lsst.multi:170 [INFO] [multi] job_id 17096792 proc_lsst.multi INFO: [multi] job_id 17096792 2024-03-16 00:06:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096792 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096792 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17096792 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-16 00:06:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-16 00:06:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:17 proc_lsst.multi:170 [INFO] [multi] job_id 17096793 proc_lsst.multi INFO: [multi] job_id 17096793 2024-03-16 00:06:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096793 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096793 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17096793 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 41, active_tasks = 184 parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 1/40 running/pending blocks, and 30 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 41 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 5 more blocks parsl.executors.status_handling INFO: Scaling out by 5 blocks parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-16 00:06:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:17 proc_lsst.multi:170 [INFO] [multi] job_id 17096794 proc_lsst.multi INFO: [multi] job_id 17096794 2024-03-16 00:06:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:06:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096794 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096794 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17096794 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-16 00:06:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:17 proc_lsst.multi:170 [INFO] [multi] job_id 17096795 proc_lsst.multi INFO: [multi] job_id 17096795 2024-03-16 00:06:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:06:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096795 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096795 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17096795 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-16 00:06:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-16 00:06:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:18 proc_lsst.multi:170 [INFO] [multi] job_id 17096796 proc_lsst.multi INFO: [multi] job_id 17096796 2024-03-16 00:06:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:06:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096796 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096796 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17096796 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-16 00:06:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:18 proc_lsst.multi:170 [INFO] [multi] job_id 17096797 proc_lsst.multi INFO: [multi] job_id 17096797 2024-03-16 00:06:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:06:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096797 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096797 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17096797 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-16 00:06:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-16 00:06:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-16 00:06:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-16 00:06:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 169.254.95.120,10.64.129.9,n3009,10.64.65.9,198.48.92.26,127.0.0.1 -p 0 -c 1.0 -m None --poll 10 --task_port=54513 --result_port=54461 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-16 00:06:19 proc_lsst.multi:170 [INFO] [multi] job_id 17096798 proc_lsst.multi INFO: [multi] job_id 17096798 2024-03-16 00:06:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-16 00:06:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-16 00:06:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17096798 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17096798 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17096798 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b026f5880885031ca5adf9a198da3989 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/33be11e6-cfb0-4eb9-b40a-a813f4bb6859_isrForCrosstalkSources_999144_11.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/33be11e6-cfb0-4eb9-b40a-a813f4bb6859_isrForCrosstalkSources_999144_11.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57460911f83a3dc144ea37c57ce63adb with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/15620bd4-036a-4c40-95ac-b8aef504942a_isrForCrosstalkSources_999143_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/15620bd4-036a-4c40-95ac-b8aef504942a_isrForCrosstalkSources_999143_28.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 182 parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/45 running/pending blocks, and 44 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f113335ba7c0f05b742387621668c287 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4c826a04-e794-4f8b-b1f2-9636555f2e19_isrForCrosstalkSources_999143_41.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4c826a04-e794-4f8b-b1f2-9636555f2e19_isrForCrosstalkSources_999143_41.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 181 parsl.dataflow.strategy DEBUG: Executor multi has 181 active tasks, 1/45 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dd5a103400f01667ea627700a2f04135 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/56eb1580-9a7e-48b7-a134-61189f96ec5e_isrForCrosstalkSources_999142_27.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/56eb1580-9a7e-48b7-a134-61189f96ec5e_isrForCrosstalkSources_999142_27.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e7cfedbb48133bf3269e82b9e336bc0 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/08dd60c6-ba6a-41dd-b3d0-a5dfc181c720_isrForCrosstalkSources_999143_26.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/08dd60c6-ba6a-41dd-b3d0-a5dfc181c720_isrForCrosstalkSources_999143_26.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 801dbddda6c3c8b8e65cf0dc9b8f1914 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/25e6e757-ddcf-4849-99a6-22cf11f6cdd7_isrForCrosstalkSources_999142_20.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/25e6e757-ddcf-4849-99a6-22cf11f6cdd7_isrForCrosstalkSources_999142_20.stderr parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e73a1a40d2549283034c943b224c072 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a7de3386-8a5f-4dcb-874f-ab59d39d8d8e_isrForCrosstalkSources_999143_21.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a7de3386-8a5f-4dcb-874f-ab59d39d8d8e_isrForCrosstalkSources_999143_21.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b912c00e7c5e25cf01294fa6c8956629 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0fbdb91-72a6-462e-9d64-a8e40bd67c91_isrForCrosstalkSources_999142_22.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0fbdb91-72a6-462e-9d64-a8e40bd67c91_isrForCrosstalkSources_999142_22.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0edfbcf57d22522d11291978855728aa with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b768824d-0760-451d-b813-c3cde9ca2b0d_isrForCrosstalkSources_999142_18.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b768824d-0760-451d-b813-c3cde9ca2b0d_isrForCrosstalkSources_999142_18.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e075ddfc701c57ab3acad3587fc7a4cb with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b967fee6-7f11-41df-afa9-ac931f9ff504_isrForCrosstalkSources_999142_5.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b967fee6-7f11-41df-afa9-ac931f9ff504_isrForCrosstalkSources_999142_5.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a51478ab3aafc4731d134a24699090 with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31a4cb9d-38a2-4168-9caa-f4b1ab9276d0_isrForCrosstalkSources_999144_21.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31a4cb9d-38a2-4168-9caa-f4b1ab9276d0_isrForCrosstalkSources_999144_21.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2feea0f0d62bfc8974201b5978f103 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe00a7c-17ee-4b8a-8f8d-9b03207cdf9b_isrForCrosstalkSources_999143_31.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe00a7c-17ee-4b8a-8f8d-9b03207cdf9b_isrForCrosstalkSources_999143_31.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 235d634e7dfc122653591d660a42f29d with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/857494ed-2157-4e16-9073-ec534424314d_isrForCrosstalkSources_999144_54.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/857494ed-2157-4e16-9073-ec534424314d_isrForCrosstalkSources_999144_54.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5417ac050a3bf20d0ca8c21dd989ecfe with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/05957d3f-683a-4893-b8aa-e1eec262a5ee_isrForCrosstalkSources_999144_49.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/05957d3f-683a-4893-b8aa-e1eec262a5ee_isrForCrosstalkSources_999144_49.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5a2d15f92e3a143070c1694e7aded33 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8b4457f0-dbcc-47a2-a739-29a43badd961_isrForCrosstalkSources_999144_14.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8b4457f0-dbcc-47a2-a739-29a43badd961_isrForCrosstalkSources_999144_14.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d2bb5eccb951c56327724b9a186b129 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f33dd437-13b5-4dff-a9b5-890458a5fc33_isrForCrosstalkSources_999143_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f33dd437-13b5-4dff-a9b5-890458a5fc33_isrForCrosstalkSources_999143_9.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6786977a5b8ec9308168c2280c04c48 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ef2316d1-f4f9-4e64-b721-8a0ff7c9f934_isrForCrosstalkSources_999143_6.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ef2316d1-f4f9-4e64-b721-8a0ff7c9f934_isrForCrosstalkSources_999143_6.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b202080ae0ee2adcc6a7444881c09f28 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/143e63f9-d122-41b4-8d10-f2d2dcc734ca_isrForCrosstalkSources_999142_53.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/143e63f9-d122-41b4-8d10-f2d2dcc734ca_isrForCrosstalkSources_999142_53.stderr 2024-03-16 00:06:32 proc_lsst.multi:146 [INFO] found job 17843 in provider local proc_lsst.multi INFO: found job 17843 in provider local 2024-03-16 00:06:32 proc_lsst.multi:146 [INFO] found job 17096750 in provider astro proc_lsst.multi INFO: found job 17096750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b6c66e4f51512e189ba94815ccc2c21 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5990d00c-d4fc-4112-b2dd-c9811f14b5dd_isrForCrosstalkSources_999144_8.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5990d00c-d4fc-4112-b2dd-c9811f14b5dd_isrForCrosstalkSources_999144_8.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c4dea592c69185c05e20a3a2e496310 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/3cd35212-59a5-4a18-bc1c-13c2fad8f167_isrForCrosstalkSources_999142_34.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/3cd35212-59a5-4a18-bc1c-13c2fad8f167_isrForCrosstalkSources_999142_34.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry babe608a3b5706e2704950a3274a997b with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d9b85a17-d257-47fa-8f3b-4daee87dd233_isrForCrosstalkSources_999143_39.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d9b85a17-d257-47fa-8f3b-4daee87dd233_isrForCrosstalkSources_999143_39.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21e6078b0bcc22fe9a46fb627e4565eb with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5c129f7-5862-41d0-8343-0bbe04c0b19c_isrForCrosstalkSources_999142_13.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5c129f7-5862-41d0-8343-0bbe04c0b19c_isrForCrosstalkSources_999142_13.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a82ee1e185c12c4538dbdaa9dde9a2c with result from task 29 parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:32 proc_lsst.multi:146 [INFO] found job 17096751 in provider astro proc_lsst.multi INFO: found job 17096751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a2bee8f4-b720-4b38-9086-e35e8b68f688_isrForCrosstalkSources_999143_56.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/a2bee8f4-b720-4b38-9086-e35e8b68f688_isrForCrosstalkSources_999143_56.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f6927c4bcf6bda94c72e665280dbbf2 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/dde246f8-4246-468c-849e-122f0e5993a6_isrForCrosstalkSources_999143_15.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/dde246f8-4246-468c-849e-122f0e5993a6_isrForCrosstalkSources_999143_15.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f9e02bd4aa01ddf13d913461fcfda78 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/403878c4-5f03-40d5-bfb3-37782c514db6_isrForCrosstalkSources_999143_10.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/403878c4-5f03-40d5-bfb3-37782c514db6_isrForCrosstalkSources_999143_10.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 335df7f5ef319c800a4c209a6b0591ca with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3dac1764-961f-4676-863e-27504defdc99_isrForCrosstalkSources_999143_11.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3dac1764-961f-4676-863e-27504defdc99_isrForCrosstalkSources_999143_11.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a49aedd0f2ef88d856c13742cd5af800 with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e0a54420-7a36-4fa3-8965-7354c7460ea3_isrForCrosstalkSources_999143_2.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e0a54420-7a36-4fa3-8965-7354c7460ea3_isrForCrosstalkSources_999143_2.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b53d414a3c1153487d382596e29deca with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/01a02d87-8b4c-434b-8f40-0b5268cf4e02_isrForCrosstalkSources_999142_7.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/01a02d87-8b4c-434b-8f40-0b5268cf4e02_isrForCrosstalkSources_999142_7.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2910b5eaf359d5045073c0ed1fedaf46 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31906b93-8b7a-43f7-9caf-1b0ff79d2aad_isrForCrosstalkSources_999144_51.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/31906b93-8b7a-43f7-9caf-1b0ff79d2aad_isrForCrosstalkSources_999144_51.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c063afe39960822f5ba9b32eba05c5e with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/b90c1139-7970-4151-ad50-43af8ecda18e_isrForCrosstalkSources_999144_62.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/b90c1139-7970-4151-ad50-43af8ecda18e_isrForCrosstalkSources_999144_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:33 proc_lsst.multi:146 [INFO] found job 17096752 in provider astro proc_lsst.multi INFO: found job 17096752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f7adc5384ffd44c023549b17763bf3f with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a0fe53bd-435a-4f09-9403-89b19d26bdc3_isrForCrosstalkSources_999144_30.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a0fe53bd-435a-4f09-9403-89b19d26bdc3_isrForCrosstalkSources_999144_30.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcae3db2b9562f7176de59eb7cb4413c with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8c36fe52-b82a-4f15-a887-f76db2c7f36f_isrForCrosstalkSources_999143_57.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8c36fe52-b82a-4f15-a887-f76db2c7f36f_isrForCrosstalkSources_999143_57.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67eaa0a1b5c515a10c3521bb6432ea34 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b86d2ac3-d0c1-4825-aba1-6fadb7373918_isrForCrosstalkSources_999142_44.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b86d2ac3-d0c1-4825-aba1-6fadb7373918_isrForCrosstalkSources_999142_44.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ee2a49ba27ba95a47e014111dde7c6d with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/51dbd366-5440-43ed-a053-7cd9910072c0_isrForCrosstalkSources_999142_52.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/51dbd366-5440-43ed-a053-7cd9910072c0_isrForCrosstalkSources_999142_52.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5efb540a52d957b2c4cc1db1871cfba with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/49d4ef39-434a-4a00-b5d4-07a5271d8e20_isrForCrosstalkSources_999143_49.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/49d4ef39-434a-4a00-b5d4-07a5271d8e20_isrForCrosstalkSources_999143_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:33 proc_lsst.multi:146 [INFO] found job 17096753 in provider astro proc_lsst.multi INFO: found job 17096753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:33 proc_lsst.multi:146 [INFO] found job 17096754 in provider astro proc_lsst.multi INFO: found job 17096754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ba997bb29db2c69751e4fe308163e3d with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5c786791-767c-46fc-9fa3-88e6e60bc4a8_isrForCrosstalkSources_999142_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5c786791-767c-46fc-9fa3-88e6e60bc4a8_isrForCrosstalkSources_999142_15.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7789055d87315b276df243d72d208d85 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1a8f9228-d12f-4559-898c-9fb440a193d5_isrForCrosstalkSources_999142_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1a8f9228-d12f-4559-898c-9fb440a193d5_isrForCrosstalkSources_999142_33.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f05057931b2db25e8ad45e865a3d0406 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/cc0a19b4-ff79-4398-abd7-abc6cd7b93b6_isrForCrosstalkSources_999142_17.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/cc0a19b4-ff79-4398-abd7-abc6cd7b93b6_isrForCrosstalkSources_999142_17.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fdbccdfe90689a9a104f08aa58bd5e3b with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9e3048ac-2a66-4bdc-a146-2e6cafc45787_isrForCrosstalkSources_999142_11.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9e3048ac-2a66-4bdc-a146-2e6cafc45787_isrForCrosstalkSources_999142_11.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4e6ad2704700968bcee683aa104a115 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/132f5606-5df2-4b3e-9c2e-b24043f260aa_isrForCrosstalkSources_999142_29.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/132f5606-5df2-4b3e-9c2e-b24043f260aa_isrForCrosstalkSources_999142_29.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df8916fad07b5c6ba6df7c50b34ddd47 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b804d082-8038-4156-bbc1-9e62d9ab8fbd_isrForCrosstalkSources_999142_47.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b804d082-8038-4156-bbc1-9e62d9ab8fbd_isrForCrosstalkSources_999142_47.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3be046635b01f17b8a26a9f3b4226180 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/849743a7-3df1-4ac3-94cb-33a08202bc51_isrForCrosstalkSources_999142_54.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/849743a7-3df1-4ac3-94cb-33a08202bc51_isrForCrosstalkSources_999142_54.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f665d306bcc23c3e32f9feaaf3035e8 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d0cfb7cd-c77a-4614-9ea2-030e0b3205c1_isrForCrosstalkSources_999142_59.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d0cfb7cd-c77a-4614-9ea2-030e0b3205c1_isrForCrosstalkSources_999142_59.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc2211881cd62c56ef12af23ce47a861 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/71dc27e9-05a8-44c5-9ec0-63cc5be17a64_isrForCrosstalkSources_999143_30.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/71dc27e9-05a8-44c5-9ec0-63cc5be17a64_isrForCrosstalkSources_999143_30.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c387947cc37abce5a0054e803784b9d with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f0e8c79b-40c8-49c6-8989-b61826ad860b_isrForCrosstalkSources_999144_5.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f0e8c79b-40c8-49c6-8989-b61826ad860b_isrForCrosstalkSources_999144_5.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3976d510f8d0cb98c30a365490edb3ea with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/481877b2-f270-4bc2-9944-45bb66439675_isrForCrosstalkSources_999144_58.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/481877b2-f270-4bc2-9944-45bb66439675_isrForCrosstalkSources_999144_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:34 proc_lsst.multi:146 [INFO] found job 17096755 in provider astro proc_lsst.multi INFO: found job 17096755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:34 proc_lsst.multi:146 [INFO] found job 17096756 in provider astro proc_lsst.multi INFO: found job 17096756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 263d511a2e6ee1cdb4b91a3d490497fc with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/fd48f427-fea5-44ab-95ab-f77566137ce2_isrForCrosstalkSources_999144_28.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/fd48f427-fea5-44ab-95ab-f77566137ce2_isrForCrosstalkSources_999144_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:35 proc_lsst.multi:146 [INFO] found job 17096757 in provider astro proc_lsst.multi INFO: found job 17096757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf8bfe52c485111d2d819068b47fa760 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02b66c12-9050-4bb2-be20-c82db2009cea_isrForCrosstalkSources_999142_51.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02b66c12-9050-4bb2-be20-c82db2009cea_isrForCrosstalkSources_999142_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:36 proc_lsst.multi:146 [INFO] found job 17096758 in provider astro proc_lsst.multi INFO: found job 17096758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08de5e1633795066d73e563f775ac4b7 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20658dfe-61eb-4fea-b249-28aa3b061154_isrForCrosstalkSources_999144_12.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20658dfe-61eb-4fea-b249-28aa3b061154_isrForCrosstalkSources_999144_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:36 proc_lsst.multi:146 [INFO] found job 17096759 in provider astro proc_lsst.multi INFO: found job 17096759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:37 proc_lsst.multi:146 [INFO] found job 17096760 in provider astro proc_lsst.multi INFO: found job 17096760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:37 proc_lsst.multi:146 [INFO] found job 17096761 in provider astro proc_lsst.multi INFO: found job 17096761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 742feb358cf5c93359231e0fa5d4eb76 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3c79ee17-5be1-4c1a-b036-35b358920fd1_isrForCrosstalkSources_999144_6.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3c79ee17-5be1-4c1a-b036-35b358920fd1_isrForCrosstalkSources_999144_6.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5987bdb0a88619b06bcdb0b064674ffa with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/82af21f0-b883-4579-af24-b5d72fa138ca_isrForCrosstalkSources_999144_29.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/82af21f0-b883-4579-af24-b5d72fa138ca_isrForCrosstalkSources_999144_29.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6f2898cfa0e8f244004e1b5a6ffef50 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/c2770ab7-e27b-4408-a2f9-84942bfac642_isrForCrosstalkSources_999144_17.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/c2770ab7-e27b-4408-a2f9-84942bfac642_isrForCrosstalkSources_999144_17.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80bb9f65141f30284144d7f400288a51 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20770d34-d29d-44e9-8705-2ec102a44365_isrForCrosstalkSources_999144_38.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/20770d34-d29d-44e9-8705-2ec102a44365_isrForCrosstalkSources_999144_38.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73fdcc70d8f2349da56e4e62dcf09d1f with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/914a11f8-d607-48bf-83f5-ae5325f64495_isrForCrosstalkSources_999144_55.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/914a11f8-d607-48bf-83f5-ae5325f64495_isrForCrosstalkSources_999144_55.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbb59bf110ae2c415a0708a87ee7decf with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/03e86b34-b987-42b9-b83a-beee60df34ed_isrForCrosstalkSources_999144_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/03e86b34-b987-42b9-b83a-beee60df34ed_isrForCrosstalkSources_999144_45.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31a4456558531a57e9b9dabeada5d7f7 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b45411e2-5e7c-4c09-b224-60023987e844_isrForCrosstalkSources_999143_14.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b45411e2-5e7c-4c09-b224-60023987e844_isrForCrosstalkSources_999143_14.stderr parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da86c66ef475ad9e18b05771fc1d6b7a with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/04309b92-a5b6-421b-8522-c6ca73944bea_isrForCrosstalkSources_999143_13.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/04309b92-a5b6-421b-8522-c6ca73944bea_isrForCrosstalkSources_999143_13.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49e370880a844642d717ad44a3ce3573 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4a88a7a6-dc2c-4e8d-b9c9-a5f5031a3847_isrForCrosstalkSources_999142_36.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4a88a7a6-dc2c-4e8d-b9c9-a5f5031a3847_isrForCrosstalkSources_999142_36.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39ad49cae1811e81ec40c34d49c921a9 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/dfc2b17e-7076-4507-a61e-12202622ce4c_isrForCrosstalkSources_999142_25.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/dfc2b17e-7076-4507-a61e-12202622ce4c_isrForCrosstalkSources_999142_25.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e3514ff95e073e09e62242e94058005 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/df5eefc9-aded-4fab-bcaa-b24b255dad28_isrForCrosstalkSources_999142_28.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/df5eefc9-aded-4fab-bcaa-b24b255dad28_isrForCrosstalkSources_999142_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:37 proc_lsst.multi:146 [INFO] found job 17096762 in provider astro proc_lsst.multi INFO: found job 17096762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:38 proc_lsst.multi:146 [INFO] found job 17096763 in provider astro proc_lsst.multi INFO: found job 17096763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:38 proc_lsst.multi:146 [INFO] found job 17096764 in provider astro proc_lsst.multi INFO: found job 17096764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:39 proc_lsst.multi:146 [INFO] found job 17096765 in provider astro proc_lsst.multi INFO: found job 17096765 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:40 proc_lsst.multi:146 [INFO] found job 17096766 in provider astro proc_lsst.multi INFO: found job 17096766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98722e87b6ff9b068ea959e3afc30b44 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77ee0de7-e749-4ced-88e7-307e829f2940_isrForCrosstalkSources_999144_1.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77ee0de7-e749-4ced-88e7-307e829f2940_isrForCrosstalkSources_999144_1.stderr parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9716fad1565650109d06d94b495b788 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4e7c5106-2abe-49cf-9d05-0bf78dfac037_isrForCrosstalkSources_999144_3.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4e7c5106-2abe-49cf-9d05-0bf78dfac037_isrForCrosstalkSources_999144_3.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b00ab0e79c1b0dc1c5bee4cb2fee6e9d with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3615f9e4-051c-4569-baed-c82a0e4d7d75_isrForCrosstalkSources_999144_24.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3615f9e4-051c-4569-baed-c82a0e4d7d75_isrForCrosstalkSources_999144_24.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 206062cd4e2568c403258a5218c031cf with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/298848ab-45f0-4775-b994-269b21e53114_isrForCrosstalkSources_999144_37.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/298848ab-45f0-4775-b994-269b21e53114_isrForCrosstalkSources_999144_37.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ea1ebfe0a5244149849c72cac8bce98 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e9511e4b-f67c-4e9f-ba82-623a2c39a997_isrForCrosstalkSources_999143_29.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e9511e4b-f67c-4e9f-ba82-623a2c39a997_isrForCrosstalkSources_999143_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:40 proc_lsst.multi:146 [INFO] found job 17096767 in provider astro proc_lsst.multi INFO: found job 17096767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:41 proc_lsst.multi:146 [INFO] found job 17096768 in provider astro proc_lsst.multi INFO: found job 17096768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:41 proc_lsst.multi:146 [INFO] found job 17096769 in provider astro proc_lsst.multi INFO: found job 17096769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc5b62180412f73d350b2c3185c4d4f6 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ab7db3ed-7729-45da-b113-e22451bca9fe_isrForCrosstalkSources_999143_38.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ab7db3ed-7729-45da-b113-e22451bca9fe_isrForCrosstalkSources_999143_38.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 168476ffad48a8150ef42b69e57ad1a4 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1c9aefe1-3da2-427f-88e7-7a6369926efc_isrForCrosstalkSources_999144_23.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1c9aefe1-3da2-427f-88e7-7a6369926efc_isrForCrosstalkSources_999144_23.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f037beb7ab1ce20e26c84493e641bcf with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1d5741b7-50a2-4bfa-9a71-c074e1f50127_isrForCrosstalkSources_999144_9.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/1d5741b7-50a2-4bfa-9a71-c074e1f50127_isrForCrosstalkSources_999144_9.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72be19bba8507bcad15a359a0eefc7bb with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/30757288-0d8c-49ae-b4cb-5fe4624bf38e_isrForCrosstalkSources_999144_16.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/30757288-0d8c-49ae-b4cb-5fe4624bf38e_isrForCrosstalkSources_999144_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:42 proc_lsst.multi:146 [INFO] found job 17096770 in provider astro proc_lsst.multi INFO: found job 17096770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60fa5a4e90b16c4146c5165367646aac with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54605ca1-ab03-42e6-bf33-86ddfeae53be_isrForCrosstalkSources_999144_60.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54605ca1-ab03-42e6-bf33-86ddfeae53be_isrForCrosstalkSources_999144_60.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9be13aed23bb9895cb99d1078dd763c4 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ee65a551-30d0-4400-b92e-d6d58d218ad6_isrForCrosstalkSources_999143_19.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ee65a551-30d0-4400-b92e-d6d58d218ad6_isrForCrosstalkSources_999143_19.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35e2c494d82cea3eef2f3475ef7ce473 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/24ed63bd-cdf8-4760-8843-c03526fa220c_isrForCrosstalkSources_999143_46.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/24ed63bd-cdf8-4760-8843-c03526fa220c_isrForCrosstalkSources_999143_46.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aaf2de9e86d14eca4d70c98343f29b06 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8bd22da7-6936-4f45-8866-5fae596598c5_isrForCrosstalkSources_999142_30.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8bd22da7-6936-4f45-8866-5fae596598c5_isrForCrosstalkSources_999142_30.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a70f39e5ebe582848cfa079ead44b05c with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bcaeabe-6b2b-4563-b6fa-d2ec87c02fd7_isrForCrosstalkSources_999142_50.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bcaeabe-6b2b-4563-b6fa-d2ec87c02fd7_isrForCrosstalkSources_999142_50.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5203687a052de61d1f22c079163d5c4 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/26e53d73-69bd-47ec-93dc-0b9b77720b97_isrForCrosstalkSources_999142_49.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/26e53d73-69bd-47ec-93dc-0b9b77720b97_isrForCrosstalkSources_999142_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:42 proc_lsst.multi:146 [INFO] found job 17096771 in provider astro proc_lsst.multi INFO: found job 17096771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:43 proc_lsst.multi:146 [INFO] found job 17096772 in provider astro proc_lsst.multi INFO: found job 17096772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbe659d212de237f55ccd033799faceb with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/953ad453-92a3-48be-823b-17b885250ab7_isrForCrosstalkSources_999144_43.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/953ad453-92a3-48be-823b-17b885250ab7_isrForCrosstalkSources_999144_43.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18c77a386587202cf2b226523d5ccd87 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/845acd14-b5c2-466e-b147-39af6eec893e_isrForCrosstalkSources_999144_53.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/845acd14-b5c2-466e-b147-39af6eec893e_isrForCrosstalkSources_999144_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:43 proc_lsst.multi:146 [INFO] found job 17096773 in provider astro proc_lsst.multi INFO: found job 17096773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdc7df19e19b6e9ec21e31bac3450293 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/00658c3c-17f8-47aa-b8f3-9478a5a2c18e_isrForCrosstalkSources_999144_32.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/00658c3c-17f8-47aa-b8f3-9478a5a2c18e_isrForCrosstalkSources_999144_32.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac279815c9213524cf069f7bbedd1d07 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/554370a1-66fb-427d-8f79-c2f031fc44d8_isrForCrosstalkSources_999144_48.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/554370a1-66fb-427d-8f79-c2f031fc44d8_isrForCrosstalkSources_999144_48.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1759e0b946bd488771b5b24f50f5d1f with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7e263e5b-efed-49b0-b389-26a25f2dbb47_isrForCrosstalkSources_999144_20.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7e263e5b-efed-49b0-b389-26a25f2dbb47_isrForCrosstalkSources_999144_20.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cefbaa161505d2dfeb77f951bc80c96e with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d5bc3116-0270-45a5-9bef-56fa4f04a984_isrForCrosstalkSources_999144_46.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d5bc3116-0270-45a5-9bef-56fa4f04a984_isrForCrosstalkSources_999144_46.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf173e7402d3a11fcb5436580a4b218f with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7eb08208-8793-4086-acbe-7860f85e5a91_isrForCrosstalkSources_999144_7.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/7eb08208-8793-4086-acbe-7860f85e5a91_isrForCrosstalkSources_999144_7.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d0b1af7d90b988a96c0e68d27df3ff2 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44a947a9-46ad-4687-b1fe-d04e7c27b28d_isrForCrosstalkSources_999142_45.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44a947a9-46ad-4687-b1fe-d04e7c27b28d_isrForCrosstalkSources_999142_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:44 proc_lsst.multi:146 [INFO] found job 17096774 in provider astro proc_lsst.multi INFO: found job 17096774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e707f3226cf89d752924d536ae78edb with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f81b5bb7-25f0-48b8-97cb-79c946440aef_isrForCrosstalkSources_999142_10.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f81b5bb7-25f0-48b8-97cb-79c946440aef_isrForCrosstalkSources_999142_10.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bee880a9654d7cd2f03d9a8a25943be with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6b3575e0-f48f-4773-9d0c-bcaf0be2ad3b_isrForCrosstalkSources_999143_40.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6b3575e0-f48f-4773-9d0c-bcaf0be2ad3b_isrForCrosstalkSources_999143_40.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d106f44199b34dcc0d523d1ccebb4cb1 with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/33f43d56-f44c-4b89-b4ac-69c8642412a4_isrForCrosstalkSources_999142_16.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/33f43d56-f44c-4b89-b4ac-69c8642412a4_isrForCrosstalkSources_999142_16.stderr parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9c8b336c4c889e9cada537cf7d70844 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/eea033f8-3954-46dd-9e06-02ed65860dd2_isrForCrosstalkSources_999143_3.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/eea033f8-3954-46dd-9e06-02ed65860dd2_isrForCrosstalkSources_999143_3.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 586344d8cfaa0b224ac06de3abe17db5 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/edefcdcd-caf8-4ba0-8f85-2adf26d87da7_isrForCrosstalkSources_999143_36.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/edefcdcd-caf8-4ba0-8f85-2adf26d87da7_isrForCrosstalkSources_999143_36.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13a488eb70329076e668c5d86a27a7cf with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/12b5982d-3d81-48b7-9501-eb0e8c3f911f_isrForCrosstalkSources_999142_31.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/12b5982d-3d81-48b7-9501-eb0e8c3f911f_isrForCrosstalkSources_999142_31.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2359e42a296bbe45c0f9d9201379b9b9 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c92ec5b9-5b71-4bd9-abf0-b39ffa8b27d9_isrForCrosstalkSources_999143_50.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c92ec5b9-5b71-4bd9-abf0-b39ffa8b27d9_isrForCrosstalkSources_999143_50.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49d2868292e0ed72f36fffd1f55de283 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/cecceb18-0902-4471-9bbc-31b9b6d71d2c_isrForCrosstalkSources_999143_53.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/cecceb18-0902-4471-9bbc-31b9b6d71d2c_isrForCrosstalkSources_999143_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:44 proc_lsst.multi:146 [INFO] found job 17096775 in provider astro proc_lsst.multi INFO: found job 17096775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c155c0229f879260d9ca309a89217794 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4d7e609d-8e27-4d78-9c6a-24bd886cb746_isrForCrosstalkSources_999144_33.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/4d7e609d-8e27-4d78-9c6a-24bd886cb746_isrForCrosstalkSources_999144_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:45 proc_lsst.multi:146 [INFO] found job 17096776 in provider astro proc_lsst.multi INFO: found job 17096776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:45 proc_lsst.multi:146 [INFO] found job 17096777 in provider astro proc_lsst.multi INFO: found job 17096777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e3e59df1871f24ab8211741ebab83bf with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3050de73-8be0-4b23-9ff7-cd1a25468413_isrForCrosstalkSources_999143_45.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/3050de73-8be0-4b23-9ff7-cd1a25468413_isrForCrosstalkSources_999143_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:46 proc_lsst.multi:146 [INFO] found job 17096778 in provider astro proc_lsst.multi INFO: found job 17096778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:46 proc_lsst.multi:146 [INFO] found job 17096779 in provider astro proc_lsst.multi INFO: found job 17096779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9dfeb5638d64dfc7b4a8993bdcdbcea7 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fddcfb40-01a2-4a11-804a-e7995684f24c_isrForCrosstalkSources_999142_43.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fddcfb40-01a2-4a11-804a-e7995684f24c_isrForCrosstalkSources_999142_43.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa0573153f7646040173c69e56299ff5 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4cfdea54-ac79-4208-881d-2b541502b4c3_isrForCrosstalkSources_999142_35.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/4cfdea54-ac79-4208-881d-2b541502b4c3_isrForCrosstalkSources_999142_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:47 proc_lsst.multi:146 [INFO] found job 17096780 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c32659cbf2e8720425752031d776360a with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a5f902d5-157a-409d-b7ea-3138dabb2c0c_isrForCrosstalkSources_999144_39.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a5f902d5-157a-409d-b7ea-3138dabb2c0c_isrForCrosstalkSources_999144_39.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f6933e7b7fc2a835a9e3473244cfd5e with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/253ca2b5-89e7-4c07-a566-eae92bb29cf2_isrForCrosstalkSources_999144_19.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/253ca2b5-89e7-4c07-a566-eae92bb29cf2_isrForCrosstalkSources_999144_19.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adb2b6a7f47006d65bf2c533b461cc1a with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4afdd38f-b874-4252-9383-317feda0c28f_isrForCrosstalkSources_999143_16.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4afdd38f-b874-4252-9383-317feda0c28f_isrForCrosstalkSources_999143_16.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96055f2894198132f40dedb338b7a6ae with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d604db69-14dd-4376-8d21-c3d71430e94b_isrForCrosstalkSources_999142_38.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d604db69-14dd-4376-8d21-c3d71430e94b_isrForCrosstalkSources_999142_38.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb4d3a1ecd13a5700133b466879ce79c with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5fae907e-2c3c-4638-bd48-181d407cf95f_isrForCrosstalkSources_999143_8.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5fae907e-2c3c-4638-bd48-181d407cf95f_isrForCrosstalkSources_999143_8.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c05aacefa875ec9250803facf9f89d31 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f3f2a3f9-e3eb-491c-ac62-210bf6de780b_isrForCrosstalkSources_999144_31.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f3f2a3f9-e3eb-491c-ac62-210bf6de780b_isrForCrosstalkSources_999144_31.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a96dfd4390584b7075d0a06609e9dcaa with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5a5150c1-9640-459e-b3d5-05520279001e_isrForCrosstalkSources_999143_35.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/5a5150c1-9640-459e-b3d5-05520279001e_isrForCrosstalkSources_999143_35.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a09927189cfdd5b639f94dfd6ad26f9 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0ca6fbc1-8019-4600-b3c5-55821708c334_isrForCrosstalkSources_999142_60.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0ca6fbc1-8019-4600-b3c5-55821708c334_isrForCrosstalkSources_999142_60.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfef5d70f0c51b61df9b34e43f1bf549 with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54ff4baf-d2d1-40f4-a01d-6e89d8996b68_isrForCrosstalkSources_999144_52.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/54ff4baf-d2d1-40f4-a01d-6e89d8996b68_isrForCrosstalkSources_999144_52.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acd75d90f041900098a6ed4c80e19929 with result from task 104 parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:47 proc_lsst.multi:146 [INFO] found job 17096781 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b968394c-4330-464e-a551-67707783f694_isrForCrosstalkSources_999143_60.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b968394c-4330-464e-a551-67707783f694_isrForCrosstalkSources_999143_60.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70e49565d9d4f0388e9dd20ca0f24ed7 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e06fa877-50d7-4d29-82e1-2dff75ed09c2_isrForCrosstalkSources_999144_40.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e06fa877-50d7-4d29-82e1-2dff75ed09c2_isrForCrosstalkSources_999144_40.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47eda74424f0c21a1b0080fb8295f0e4 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/80ae7cc8-84ea-4d20-bd60-f57a64f595f8_isrForCrosstalkSources_999143_1.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/80ae7cc8-84ea-4d20-bd60-f57a64f595f8_isrForCrosstalkSources_999143_1.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 935f1b8ddc360b21ffdcdda2ddbba0cc with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a86461bd-6a78-4bb5-a46e-b3c00091a0d6_isrForCrosstalkSources_999144_44.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/a86461bd-6a78-4bb5-a46e-b3c00091a0d6_isrForCrosstalkSources_999144_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:48 proc_lsst.multi:146 [INFO] found job 17096782 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:49 proc_lsst.multi:146 [INFO] found job 17096787 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096787 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33dcbb7d31bc6577d030595803a9dec3 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9570c945-ae54-48e2-93ae-3d0f0865ca0a_isrForCrosstalkSources_999142_32.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9570c945-ae54-48e2-93ae-3d0f0865ca0a_isrForCrosstalkSources_999142_32.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18615298705f706109ccd6d7d5d45eb2 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dedce3e-1b09-4027-b116-4dc544e3a9f2_isrForCrosstalkSources_999142_56.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dedce3e-1b09-4027-b116-4dc544e3a9f2_isrForCrosstalkSources_999142_56.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1c340cd985f23a11ed99e099ff2492a with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1e18a3-3a89-455c-9a15-a3e0422a5653_isrForCrosstalkSources_999142_12.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1e18a3-3a89-455c-9a15-a3e0422a5653_isrForCrosstalkSources_999142_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:49 proc_lsst.multi:146 [INFO] found job 17096788 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096788 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af05d20e6edc7531ab513bae0a735db4 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/020c3b8a-7140-49d8-9019-cfac0065f683_isrForCrosstalkSources_999142_2.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/020c3b8a-7140-49d8-9019-cfac0065f683_isrForCrosstalkSources_999142_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:49 proc_lsst.multi:146 [INFO] found job 17096789 in provider escience_a40 proc_lsst.multi INFO: found job 17096789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:49 proc_lsst.multi:146 [INFO] found job 17096790 in provider escience_a40 proc_lsst.multi INFO: found job 17096790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:50 proc_lsst.multi:146 [INFO] found job 17096791 in provider escience_a40 proc_lsst.multi INFO: found job 17096791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:50 proc_lsst.multi:146 [INFO] found job 17096792 in provider escience_a40 proc_lsst.multi INFO: found job 17096792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fd3f4b0c5a2b49616857857640a92c3 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dc4e906-1837-4aa8-87b5-fad696e5d9ef_isrForCrosstalkSources_999142_6.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5dc4e906-1837-4aa8-87b5-fad696e5d9ef_isrForCrosstalkSources_999142_6.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9eaf6a7c29615b7c7fc57e4323744469 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b2067b94-7b05-4fa7-ab48-093d3ded6fe1_isrForCrosstalkSources_999142_24.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/b2067b94-7b05-4fa7-ab48-093d3ded6fe1_isrForCrosstalkSources_999142_24.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03f1d0561e5efe474724f57e4911f595 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9ad3772c-4e36-49e5-82f6-e9a16ad3800f_isrForCrosstalkSources_999143_20.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9ad3772c-4e36-49e5-82f6-e9a16ad3800f_isrForCrosstalkSources_999143_20.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry befbd47659390b6e07250d0430d93abf with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ac25ebff-24f8-4938-ad07-89a16a1d1d5c_isrForCrosstalkSources_999143_42.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/ac25ebff-24f8-4938-ad07-89a16a1d1d5c_isrForCrosstalkSources_999143_42.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9937eeb7fcd2cb2cd57faad8fc817d49 with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f5882ca2-7ecd-4091-a235-bf755b78b69a_isrForCrosstalkSources_999143_37.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f5882ca2-7ecd-4091-a235-bf755b78b69a_isrForCrosstalkSources_999143_37.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce580030da6870a2cc8aa3994ca727bb with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9c1ced00-0723-48bb-b041-772cfa004566_isrForCrosstalkSources_999142_41.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/9c1ced00-0723-48bb-b041-772cfa004566_isrForCrosstalkSources_999142_41.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f328ed48691535ba4585a348a0770715 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8606a537-0712-441d-aa47-31c427fcea5a_isrForCrosstalkSources_999144_2.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8606a537-0712-441d-aa47-31c427fcea5a_isrForCrosstalkSources_999144_2.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 752ee98fcf27df5b108215a93b9a8a44 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d171a9a6-282e-426a-8e60-2eeb30b765e3_isrForCrosstalkSources_999142_21.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/d171a9a6-282e-426a-8e60-2eeb30b765e3_isrForCrosstalkSources_999142_21.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae577de97ac2904582757a66fba9de2e with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/94e3f774-a75e-4398-bcf3-c5ec17d27681_isrForCrosstalkSources_999144_10.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/94e3f774-a75e-4398-bcf3-c5ec17d27681_isrForCrosstalkSources_999144_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:50 proc_lsst.multi:146 [INFO] found job 17096793 in provider escience_a40 proc_lsst.multi INFO: found job 17096793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ea107bca0e38d3d2a10d27628763928 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5e7186c-e7fb-433c-9a1a-2bb2ff1af54b_isrForCrosstalkSources_999142_61.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f5e7186c-e7fb-433c-9a1a-2bb2ff1af54b_isrForCrosstalkSources_999142_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:51 proc_lsst.multi:146 [INFO] found job 17096794 in provider escience_a40 proc_lsst.multi INFO: found job 17096794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:52 proc_lsst.multi:146 [INFO] found job 17096795 in provider escience_a40 proc_lsst.multi INFO: found job 17096795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:52 proc_lsst.multi:146 [INFO] found job 17096796 in provider escience_a40 proc_lsst.multi INFO: found job 17096796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:53 proc_lsst.multi:146 [INFO] found job 17096797 in provider escience_a40 proc_lsst.multi INFO: found job 17096797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:06:53 proc_lsst.multi:146 [INFO] found job 17096798 in provider escience_a40 proc_lsst.multi INFO: found job 17096798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23a1207d719d651e0ed90a8ba5e27350 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/082a6cfa-d11e-46fb-9fdb-66af283667b9_isrForCrosstalkSources_999143_55.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/082a6cfa-d11e-46fb-9fdb-66af283667b9_isrForCrosstalkSources_999143_55.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b9c59f6fd3eaf93913fcb5cff768dd with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5925ff58-1736-45c7-889a-538ef4a8e548_isrForCrosstalkSources_999144_22.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/5925ff58-1736-45c7-889a-538ef4a8e548_isrForCrosstalkSources_999144_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 60 parsl.dataflow.strategy DEBUG: Executor multi has 60 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 60 parsl.dataflow.strategy DEBUG: Executor multi has 60 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a796d89276ce7075625fa06b78832411 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f43c0444-5e2a-47ad-86a1-fb4dec1e31c6_isrForCrosstalkSources_999143_54.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f43c0444-5e2a-47ad-86a1-fb4dec1e31c6_isrForCrosstalkSources_999143_54.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d32c89cdab19aa053f5f51acced1d731 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/924c30cb-cd12-4bf0-bfaf-59b03cf1b9c7_isrForCrosstalkSources_999143_23.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/924c30cb-cd12-4bf0-bfaf-59b03cf1b9c7_isrForCrosstalkSources_999143_23.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19157c463d680ab86babbbe2280a1e10 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/87e1d534-7446-400f-a1d9-7d3bebf86ed4_isrForCrosstalkSources_999143_43.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/87e1d534-7446-400f-a1d9-7d3bebf86ed4_isrForCrosstalkSources_999143_43.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7bf17ee0ac2113d047fbf4997086e1e with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/adea87f4-e295-4c51-8693-9e1e550e72c3_isrForCrosstalkSources_999143_59.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/adea87f4-e295-4c51-8693-9e1e550e72c3_isrForCrosstalkSources_999143_59.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4643718452293de5321e63fa391a0975 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0f3b12dc-cea3-4e65-a3c6-481b194d9c57_isrForCrosstalkSources_999143_32.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0f3b12dc-cea3-4e65-a3c6-481b194d9c57_isrForCrosstalkSources_999143_32.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad0c24b4d1a2d6e1aec9caed3057773 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0cb31b1b-18c5-4f2c-8302-1ce59eaeee20_isrForCrosstalkSources_999143_33.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/0cb31b1b-18c5-4f2c-8302-1ce59eaeee20_isrForCrosstalkSources_999143_33.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 023904a7ee9e32318a99aa6f83cdc889 with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f2ff2acc-eb8d-454b-b982-837cb63c3af3_isrForCrosstalkSources_999144_13.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f2ff2acc-eb8d-454b-b982-837cb63c3af3_isrForCrosstalkSources_999144_13.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 315839b7f9f8f810ecc386d3a96a0f2e with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/746ac36c-106e-4ebf-b6cf-8e59e9897b77_isrForCrosstalkSources_999143_51.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/746ac36c-106e-4ebf-b6cf-8e59e9897b77_isrForCrosstalkSources_999143_51.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32a4397f74a0c619fbaefc965ff0ffe8 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c8b36c90-c604-4503-93f5-bbceeb72e348_isrForCrosstalkSources_999142_14.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c8b36c90-c604-4503-93f5-bbceeb72e348_isrForCrosstalkSources_999142_14.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef18854f3776276924746b82caf5bc17 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8231a74d-6eaf-4112-ac5a-d7442daed0e1_isrForCrosstalkSources_999142_46.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/8231a74d-6eaf-4112-ac5a-d7442daed0e1_isrForCrosstalkSources_999142_46.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ca5402a3ea2fafca97edc6fd243997e with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f402e564-10b5-4b6a-a01f-27a8eab1e3ee_isrForCrosstalkSources_999142_42.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/f402e564-10b5-4b6a-a01f-27a8eab1e3ee_isrForCrosstalkSources_999142_42.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8ac4ad2a638e9ecca9bbaaf0002f397 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02edaf65-ac25-4324-8a6b-2163fca5ce6c_isrForCrosstalkSources_999142_1.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/02edaf65-ac25-4324-8a6b-2163fca5ce6c_isrForCrosstalkSources_999142_1.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5f323ed7150bb6cadac63feb2167101 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/bc22dd85-a85e-40d5-90b6-a0431671b0bd_isrForCrosstalkSources_999143_5.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/bc22dd85-a85e-40d5-90b6-a0431671b0bd_isrForCrosstalkSources_999143_5.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bdaff7547ff21fd19904367a134f61f1 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e6e6b34a-655d-4863-a7aa-a9e4ee3cb230_isrForCrosstalkSources_999144_59.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/e6e6b34a-655d-4863-a7aa-a9e4ee3cb230_isrForCrosstalkSources_999144_59.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e5b14d4be788d6d8e5a941df73ad7ee with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/7591b490-6b59-4c93-9cdb-69eca0639aff_isrForCrosstalkSources_999142_26.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/7591b490-6b59-4c93-9cdb-69eca0639aff_isrForCrosstalkSources_999142_26.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 723a23a5af73cee7499d9044a9f61ca8 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8444bbc6-61fa-4a18-a537-d3f17844c0d3_isrForCrosstalkSources_999144_61.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/8444bbc6-61fa-4a18-a537-d3f17844c0d3_isrForCrosstalkSources_999144_61.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d2a0e4d8c8b43f6309065ce0f09b606b with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0a6cc0a7-2fad-4c14-b536-26fbb205d50b_isrForCrosstalkSources_999144_41.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0a6cc0a7-2fad-4c14-b536-26fbb205d50b_isrForCrosstalkSources_999144_41.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02fa79fe59e872c8475b46320e0e5890 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c6e55db6-e30f-48dc-a1ae-46c8e294bc8d_isrForCrosstalkSources_999143_48.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c6e55db6-e30f-48dc-a1ae-46c8e294bc8d_isrForCrosstalkSources_999143_48.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 59cef603aeb340ea6ad3a25e84929f4e with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0b2ba3c-13c8-4c17-8bfa-6ae70ab22ed4_isrForCrosstalkSources_999142_3.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/a0b2ba3c-13c8-4c17-8bfa-6ae70ab22ed4_isrForCrosstalkSources_999142_3.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48a579dd3f253e7a7fc36a423d00a209 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1ea51f-95b4-4fd2-b20a-ce0a12463d14_isrForCrosstalkSources_999142_19.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/1e1ea51f-95b4-4fd2-b20a-ce0a12463d14_isrForCrosstalkSources_999142_19.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e547829d90558c014827f603f2ae862 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/28e458fd-c8f6-4862-8331-a29e4a9e168f_isrForCrosstalkSources_999144_27.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/28e458fd-c8f6-4862-8331-a29e4a9e168f_isrForCrosstalkSources_999144_27.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3ee97c77c538897fb426c35615eeb0a5 with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d1bba610-6846-4065-9192-40c222112640_isrForCrosstalkSources_999143_44.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/d1bba610-6846-4065-9192-40c222112640_isrForCrosstalkSources_999143_44.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89c52e80b7d20fbdea26f0c4285bb5a7 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4dd0d19a-a10d-4383-bf9b-4c74d0785778_isrForCrosstalkSources_999143_7.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4dd0d19a-a10d-4383-bf9b-4c74d0785778_isrForCrosstalkSources_999143_7.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 571244be64661ed5aa54d3f6f083a121 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0a3a4145-2196-4954-8040-084b309660fb_isrForCrosstalkSources_999142_55.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/0a3a4145-2196-4954-8040-084b309660fb_isrForCrosstalkSources_999142_55.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14fb16b742bc93f3e689cbd2a34f6729 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fc8f47df-a3c7-4829-9b86-69eaf973702c_isrForCrosstalkSources_999142_57.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/fc8f47df-a3c7-4829-9b86-69eaf973702c_isrForCrosstalkSources_999142_57.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cac4cabfa24a648de629fd328650af0a with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77468676-0b7d-4fbb-8b97-33ae2fb0d1ac_isrForCrosstalkSources_999144_57.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/77468676-0b7d-4fbb-8b97-33ae2fb0d1ac_isrForCrosstalkSources_999144_57.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7983d496253f514223197d72349f696 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/edb2997e-aba5-45e7-afdf-eab98be04049_isrForCrosstalkSources_999144_25.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/edb2997e-aba5-45e7-afdf-eab98be04049_isrForCrosstalkSources_999144_25.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12c6df995d0947a7f3823ca82a68d373 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3191c074-d861-42e2-8bf7-52eb19223aa0_isrForCrosstalkSources_999144_4.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/3191c074-d861-42e2-8bf7-52eb19223aa0_isrForCrosstalkSources_999144_4.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43bf6d59c105343a2ecee985176cd24b with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8fdeb541-3d8c-48fd-96ad-d314bad773b1_isrForCrosstalkSources_999143_61.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/8fdeb541-3d8c-48fd-96ad-d314bad773b1_isrForCrosstalkSources_999143_61.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82745aa3bffcc8991c18a0ac2373e7a8 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f524fd6a-1235-4aaf-8b10-2d8eed6bfa0a_isrForCrosstalkSources_999143_12.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/f524fd6a-1235-4aaf-8b10-2d8eed6bfa0a_isrForCrosstalkSources_999143_12.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1fafadf507a92249fd88c5edde5a54f2 with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c5c18299-2ee2-4e0e-a47c-3f64f560e4eb_isrForCrosstalkSources_999142_58.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/c5c18299-2ee2-4e0e-a47c-3f64f560e4eb_isrForCrosstalkSources_999142_58.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6026762f4e5853004d6828f542cd08d7 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/492887fd-3c8e-4a72-946c-47c3757725e5_isrForCrosstalkSources_999142_62.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/492887fd-3c8e-4a72-946c-47c3757725e5_isrForCrosstalkSources_999142_62.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c3211fea43f702dac3a4ffd48624653 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d4ab1c1a-bd62-44df-ad22-50b4a8931a95_isrForCrosstalkSources_999144_26.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d4ab1c1a-bd62-44df-ad22-50b4a8931a95_isrForCrosstalkSources_999144_26.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f0f2780d323e19614d8d5e26aba0c4b with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/857c0452-e5b3-4142-b164-451382bfcb90_isrForCrosstalkSources_999143_24.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/857c0452-e5b3-4142-b164-451382bfcb90_isrForCrosstalkSources_999143_24.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 941a9abec23446c929d7fe3a278ca34e with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/102bcc3e-b0cb-49ee-bbbc-7022b29f3a52_isrForCrosstalkSources_999143_22.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/102bcc3e-b0cb-49ee-bbbc-7022b29f3a52_isrForCrosstalkSources_999143_22.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c43e004c18eda71db4c1e31e89eef5d8 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c8f4070a-76ae-4594-ad7c-52e0e2fadef8_isrForCrosstalkSources_999143_18.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/c8f4070a-76ae-4594-ad7c-52e0e2fadef8_isrForCrosstalkSources_999143_18.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e95f51d666c1375a7db85f9709f8689 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f46fb3c9-ae8a-4248-9420-036e2d44411d_isrForCrosstalkSources_999144_42.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/f46fb3c9-ae8a-4248-9420-036e2d44411d_isrForCrosstalkSources_999144_42.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7fe79a1addfd058cff455ba3e948ed7d with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9a777d70-0b16-4cd9-bc57-64bb85496ce0_isrForCrosstalkSources_999143_4.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9a777d70-0b16-4cd9-bc57-64bb85496ce0_isrForCrosstalkSources_999143_4.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eff8f558c77de51b4a6bc472c7e3dcf6 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d6aaab65-34aa-4594-beb7-67637e2e7af9_isrForCrosstalkSources_999144_56.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/d6aaab65-34aa-4594-beb7-67637e2e7af9_isrForCrosstalkSources_999144_56.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65d163f03adc936bcbd8f3395b378732 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/ecca6f78-2e44-4b03-bc08-24e76ed425a0_isrForCrosstalkSources_999144_47.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/ecca6f78-2e44-4b03-bc08-24e76ed425a0_isrForCrosstalkSources_999144_47.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fe4d130720c05f78df2c105893c6905 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4a842dba-3fb6-4910-be56-9f66abe93336_isrForCrosstalkSources_999143_52.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/4a842dba-3fb6-4910-be56-9f66abe93336_isrForCrosstalkSources_999143_52.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5437ebd40444605aa246032e6e0c6556 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe81a4e-7572-4ff7-9d42-fd12d378d7ee_isrForCrosstalkSources_999143_58.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/9fe81a4e-7572-4ff7-9d42-fd12d378d7ee_isrForCrosstalkSources_999143_58.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c12c426ac974efa13f1bcb17a5be91a3 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/11409320-efd1-4bf5-995b-67995e5b2026_isrForCrosstalkSources_999142_4.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/11409320-efd1-4bf5-995b-67995e5b2026_isrForCrosstalkSources_999142_4.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 17 parsl.dataflow.strategy DEBUG: Executor multi has 17 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f551d58964722b60ebe00fa5090515c with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/aaaea80d-0a1e-4873-94e2-edd7c64795d2_isrForCrosstalkSources_999142_48.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/aaaea80d-0a1e-4873-94e2-edd7c64795d2_isrForCrosstalkSources_999142_48.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d9bac80be32b0f7834c2bc25617b773 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/502e980f-a2e5-4c35-8d56-73164369effb_isrForCrosstalkSources_999144_18.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/502e980f-a2e5-4c35-8d56-73164369effb_isrForCrosstalkSources_999144_18.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3d537754504a8c7585f8a7b1efe32e4 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/1c9224ac-a1ef-4e42-b4ef-705ab8b063bd_isrForCrosstalkSources_999143_25.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/1c9224ac-a1ef-4e42-b4ef-705ab8b063bd_isrForCrosstalkSources_999143_25.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3a5c7ef824f724e5848693dc706773b with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44695da1-ed6d-4b87-a7af-1e488f4d3d8e_isrForCrosstalkSources_999142_37.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/44695da1-ed6d-4b87-a7af-1e488f4d3d8e_isrForCrosstalkSources_999142_37.stderr 2024-03-16 00:07:04 proc_lsst.multi:146 [INFO] found job 17843 in provider local proc_lsst.multi INFO: found job 17843 in provider local 2024-03-16 00:07:04 proc_lsst.multi:146 [INFO] found job 17096750 in provider astro proc_lsst.multi INFO: found job 17096750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:04 proc_lsst.multi:146 [INFO] found job 17096751 in provider astro proc_lsst.multi INFO: found job 17096751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:04 proc_lsst.multi:146 [INFO] found job 17096752 in provider astro proc_lsst.multi INFO: found job 17096752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:05 proc_lsst.multi:146 [INFO] found job 17096753 in provider astro proc_lsst.multi INFO: found job 17096753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:05 proc_lsst.multi:146 [INFO] found job 17096754 in provider astro proc_lsst.multi INFO: found job 17096754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:05 proc_lsst.multi:146 [INFO] found job 17096755 in provider astro proc_lsst.multi INFO: found job 17096755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:05 proc_lsst.multi:146 [INFO] found job 17096756 in provider astro proc_lsst.multi INFO: found job 17096756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc04e03e0351277d9ec617a5033ee338 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e4018291-1c2e-4090-b099-6f8e858d1f27_isrForCrosstalkSources_999143_47.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e4018291-1c2e-4090-b099-6f8e858d1f27_isrForCrosstalkSources_999143_47.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6872534cb32fdf52d450ba343f47ff69 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6f19dae7-746b-42de-a9b0-2905c1749ad0_isrForCrosstalkSources_999143_34.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/6f19dae7-746b-42de-a9b0-2905c1749ad0_isrForCrosstalkSources_999143_34.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6714bee537022d96a7ac05e094ade36c with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/447720a1-c4de-425b-bf38-d29a2b9ec0f7_isrForCrosstalkSources_999142_8.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/447720a1-c4de-425b-bf38-d29a2b9ec0f7_isrForCrosstalkSources_999142_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:06 proc_lsst.multi:146 [INFO] found job 17096757 in provider astro proc_lsst.multi INFO: found job 17096757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1832cced40e5b53c76e8b5d3c75ed668 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/00f8f14d-b211-48d7-b7a8-a4474df17d29_isrForCrosstalkSources_999142_23.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/00f8f14d-b211-48d7-b7a8-a4474df17d29_isrForCrosstalkSources_999142_23.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc691a3a98a01a46e86240adf4fd8276 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/df9aa7be-903f-4ef6-83d6-5f98379fbe0d_isrForCrosstalkSources_999143_17.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/df9aa7be-903f-4ef6-83d6-5f98379fbe0d_isrForCrosstalkSources_999143_17.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68b486cccc2228420e8eb948f1aa1843 with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e432a7d4-1762-4992-9237-53708d143125_isrForCrosstalkSources_999143_27.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/e432a7d4-1762-4992-9237-53708d143125_isrForCrosstalkSources_999143_27.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baef1cd603584cf44e48720a2aa2dfc2 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/73b32838-bba0-4736-8286-19ce3bbb9a25_isrForCrosstalkSources_999142_39.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/73b32838-bba0-4736-8286-19ce3bbb9a25_isrForCrosstalkSources_999142_39.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75b7b2271bacb19582a80777227cf42d with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b439e7a8-db2b-49ec-8d5d-9172cb96de5c_isrForCrosstalkSources_999143_62.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999143/b439e7a8-db2b-49ec-8d5d-9172cb96de5c_isrForCrosstalkSources_999143_62.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6b56e77ae7e128aa2cd7cdffeea25ef with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bea4b9a-0421-4f7c-b13e-188b834699fb_isrForCrosstalkSources_999142_9.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999142/5bea4b9a-0421-4f7c-b13e-188b834699fb_isrForCrosstalkSources_999142_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:06 proc_lsst.multi:146 [INFO] found job 17096758 in provider astro proc_lsst.multi INFO: found job 17096758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eead6fa25e8bd8eaee99d4eac4d2ec7 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/91950290-447b-4f31-98d8-df74e53ef041_isrForCrosstalkSources_999144_34.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/91950290-447b-4f31-98d8-df74e53ef041_isrForCrosstalkSources_999144_34.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad0f3db5506e77b5f09b7863526117fa with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0ffd96e6-401f-421d-befe-cf49108dacf5_isrForCrosstalkSources_999144_35.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/0ffd96e6-401f-421d-befe-cf49108dacf5_isrForCrosstalkSources_999144_35.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 540bc302d06b22cb867a816cfecfd158 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/07c84200-7544-4302-8538-c4e47dfe29d1_isrForCrosstalkSources_999144_36.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/07c84200-7544-4302-8538-c4e47dfe29d1_isrForCrosstalkSources_999144_36.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b09db34f3e44a39de625cb73d63b6ea2 with result from task 179 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/efd21f7f-c233-4289-b29e-244de8b6d35c_isrForCrosstalkSources_999144_50.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/submit/DEEP/20210518/A1f/science#step0/20240316T000537Z/logs/isrForCrosstalkSources/999144/efd21f7f-c233-4289-b29e-244de8b6d35c_isrForCrosstalkSources_999144_50.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:06 proc_lsst.multi:146 [INFO] found job 17096759 in provider astro proc_lsst.multi INFO: found job 17096759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:06 proc_lsst.multi:146 [INFO] found job 17096760 in provider astro proc_lsst.multi INFO: found job 17096760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:07 proc_lsst.multi:146 [INFO] found job 17096761 in provider astro proc_lsst.multi INFO: found job 17096761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:07 proc_lsst.multi:146 [INFO] found job 17096762 in provider astro proc_lsst.multi INFO: found job 17096762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:07 proc_lsst.multi:146 [INFO] found job 17096763 in provider astro proc_lsst.multi INFO: found job 17096763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:07 proc_lsst.multi:146 [INFO] found job 17096764 in provider astro proc_lsst.multi INFO: found job 17096764 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:07 proc_lsst.multi:146 [INFO] found job 17096765 in provider astro proc_lsst.multi INFO: found job 17096765 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:08 proc_lsst.multi:146 [INFO] found job 17096766 in provider astro proc_lsst.multi INFO: found job 17096766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:08 proc_lsst.multi:146 [INFO] found job 17096767 in provider astro proc_lsst.multi INFO: found job 17096767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:08 proc_lsst.multi:146 [INFO] found job 17096768 in provider astro proc_lsst.multi INFO: found job 17096768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:08 proc_lsst.multi:146 [INFO] found job 17096769 in provider astro proc_lsst.multi INFO: found job 17096769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:09 proc_lsst.multi:146 [INFO] found job 17096770 in provider astro proc_lsst.multi INFO: found job 17096770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:09 proc_lsst.multi:146 [INFO] found job 17096771 in provider astro proc_lsst.multi INFO: found job 17096771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:09 proc_lsst.multi:146 [INFO] found job 17096772 in provider astro proc_lsst.multi INFO: found job 17096772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:09 proc_lsst.multi:146 [INFO] found job 17096773 in provider astro proc_lsst.multi INFO: found job 17096773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:10 proc_lsst.multi:146 [INFO] found job 17096774 in provider astro proc_lsst.multi INFO: found job 17096774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:10 proc_lsst.multi:146 [INFO] found job 17096775 in provider astro proc_lsst.multi INFO: found job 17096775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:10 proc_lsst.multi:146 [INFO] found job 17096776 in provider astro proc_lsst.multi INFO: found job 17096776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:10 proc_lsst.multi:146 [INFO] found job 17096777 in provider astro proc_lsst.multi INFO: found job 17096777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:10 proc_lsst.multi:146 [INFO] found job 17096778 in provider astro proc_lsst.multi INFO: found job 17096778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:11 proc_lsst.multi:146 [INFO] found job 17096779 in provider astro proc_lsst.multi INFO: found job 17096779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096750,17096751,17096752,17096753,17096754,17096755,17096756,17096757,17096758,17096759,17096760,17096761,17096762,17096763,17096764,17096765,17096766,17096767,17096768,17096769,17096770,17096771,17096772,17096773,17096774,17096775,17096776,17096777,17096778,17096779' parsl.providers.slurm.slurm DEBUG: squeue returned 17096778 R 17096779 R 17096773 R 17096774 R 17096775 R 17096776 R 17096777 R 17096770 R 17096771 R 17096772 R 17096769 R 17096766 R 17096767 R 17096768 R 17096760 R 17096761 R 17096762 R 17096763 R 17096764 R 17096765 R 17096756 R 17096757 R 17096758 R 17096759 R 17096753 R 17096754 R 17096755 R 17096750 R 17096751 R 17096752 R parsl.providers.slurm.slurm DEBUG: Updating job 17096778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096764 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096765 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096751 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096752 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:11 proc_lsst.multi:146 [INFO] found job 17096780 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096780 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:11 proc_lsst.multi:146 [INFO] found job 17096781 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:11 proc_lsst.multi:146 [INFO] found job 17096782 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:12 proc_lsst.multi:146 [INFO] found job 17096787 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096787 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:12 proc_lsst.multi:146 [INFO] found job 17096788 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096788 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096780,17096781,17096782,17096787,17096788' parsl.providers.slurm.slurm DEBUG: squeue returned 17096788 R 17096787 R 17096780 R 17096781 R 17096782 R parsl.providers.slurm.slurm DEBUG: Updating job 17096788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096787 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096781 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096782 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:12 proc_lsst.multi:146 [INFO] found job 17096789 in provider escience_a40 proc_lsst.multi INFO: found job 17096789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:12 proc_lsst.multi:146 [INFO] found job 17096790 in provider escience_a40 proc_lsst.multi INFO: found job 17096790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:13 proc_lsst.multi:146 [INFO] found job 17096791 in provider escience_a40 proc_lsst.multi INFO: found job 17096791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:13 proc_lsst.multi:146 [INFO] found job 17096792 in provider escience_a40 proc_lsst.multi INFO: found job 17096792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:13 proc_lsst.multi:146 [INFO] found job 17096793 in provider escience_a40 proc_lsst.multi INFO: found job 17096793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:13 proc_lsst.multi:146 [INFO] found job 17096794 in provider escience_a40 proc_lsst.multi INFO: found job 17096794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:14 proc_lsst.multi:146 [INFO] found job 17096795 in provider escience_a40 proc_lsst.multi INFO: found job 17096795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:14 proc_lsst.multi:146 [INFO] found job 17096796 in provider escience_a40 proc_lsst.multi INFO: found job 17096796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:14 proc_lsst.multi:146 [INFO] found job 17096797 in provider escience_a40 proc_lsst.multi INFO: found job 17096797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING 2024-03-16 00:07:14 proc_lsst.multi:146 [INFO] found job 17096798 in provider escience_a40 proc_lsst.multi INFO: found job 17096798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17096789,17096790,17096791,17096792,17096793,17096794,17096795,17096796,17096797,17096798' parsl.providers.slurm.slurm DEBUG: squeue returned 17096797 R 17096798 R 17096791 R 17096792 R 17096793 R 17096794 R 17096795 R 17096796 R 17096789 R 17096790 R parsl.providers.slurm.slurm DEBUG: Updating job 17096797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17096790 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 46/0 running/pending blocks, and 47 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (46) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 3.337860107421875e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22537201380624 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=46, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 46 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 58f41adadb47 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 58f41adadb47 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce84c7241adb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce84c7241adb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7ad0600071d5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7ad0600071d5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff136f83181d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff136f83181d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 85e03a829cbe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 85e03a829cbe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffaecebbec48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffaecebbec48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9b79335bc812 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9b79335bc812 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17f3834110ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17f3834110ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 65318308eba6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 65318308eba6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3565ff9f5543 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3565ff9f5543 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cfff925b5f28 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cfff925b5f28 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0611ac7ea762 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0611ac7ea762 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3e32863df73 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3e32863df73 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 18f08fdcc739 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 18f08fdcc739 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05707f6f77a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05707f6f77a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb96e10d5243 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb96e10d5243 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 05358c102528 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 05358c102528 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84d1e9b02ae3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84d1e9b02ae3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 83e30c844825 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 83e30c844825 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 845d1e83d98b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 845d1e83d98b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12bf4ebc0534 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12bf4ebc0534 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1b338e8cd773 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1b338e8cd773 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 626b83113889 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 626b83113889 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 71d9417bc470 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 71d9417bc470 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 82829b471f43 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 82829b471f43 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 586545c156d4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 586545c156d4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d28b96985157 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d28b96985157 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8ee04d04685 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8ee04d04685 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 88ab4ee41a29 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 88ab4ee41a29 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba5cd720d85f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba5cd720d85f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c20408b40948 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c20408b40948 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 85d4901b8cde parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 85d4901b8cde parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7cbed904c54f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7cbed904c54f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17de388512c4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17de388512c4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 724393ccbb6a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 724393ccbb6a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f814d8903ee9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f814d8903ee9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8e266a55d105 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8e266a55d105 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e450a8bf12dc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e450a8bf12dc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 650a668b475d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 650a668b475d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 25f2c3f4f0db parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 25f2c3f4f0db parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29ebe72e9110 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29ebe72e9110 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0472c0f2326 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0472c0f2326 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 742e1a150769 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 742e1a150769 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 934d43c3bc0d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 934d43c3bc0d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8db7c49822ca parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8db7c49822ca parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 94503299a5c0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 94503299a5c0 2024-03-16 00:07:14 proc_lsst.multi:146 [INFO] found job 17096773 in provider astro proc_lsst.multi INFO: found job 17096773 in provider astro 2024-03-16 00:07:14 proc_lsst.multi:201 [INFO] cancelling 17096773 on provider astro proc_lsst.multi INFO: cancelling 17096773 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096765 in provider astro proc_lsst.multi INFO: found job 17096765 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096765 on provider astro proc_lsst.multi INFO: cancelling 17096765 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096760 in provider astro proc_lsst.multi INFO: found job 17096760 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096760 on provider astro proc_lsst.multi INFO: cancelling 17096760 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096770 in provider astro proc_lsst.multi INFO: found job 17096770 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096770 on provider astro proc_lsst.multi INFO: cancelling 17096770 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096772 in provider astro proc_lsst.multi INFO: found job 17096772 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096772 on provider astro proc_lsst.multi INFO: cancelling 17096772 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096762 in provider astro proc_lsst.multi INFO: found job 17096762 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096762 on provider astro proc_lsst.multi INFO: cancelling 17096762 on provider astro 2024-03-16 00:07:15 proc_lsst.multi:146 [INFO] found job 17096764 in provider astro proc_lsst.multi INFO: found job 17096764 in provider astro 2024-03-16 00:07:15 proc_lsst.multi:201 [INFO] cancelling 17096764 on provider astro proc_lsst.multi INFO: cancelling 17096764 on provider astro 2024-03-16 00:07:16 proc_lsst.multi:146 [INFO] found job 17096775 in provider astro proc_lsst.multi INFO: found job 17096775 in provider astro 2024-03-16 00:07:16 proc_lsst.multi:201 [INFO] cancelling 17096775 on provider astro proc_lsst.multi INFO: cancelling 17096775 on provider astro 2024-03-16 00:07:16 proc_lsst.multi:146 [INFO] found job 17096763 in provider astro proc_lsst.multi INFO: found job 17096763 in provider astro 2024-03-16 00:07:16 proc_lsst.multi:201 [INFO] cancelling 17096763 on provider astro proc_lsst.multi INFO: cancelling 17096763 on provider astro 2024-03-16 00:07:16 proc_lsst.multi:146 [INFO] found job 17096774 in provider astro proc_lsst.multi INFO: found job 17096774 in provider astro 2024-03-16 00:07:16 proc_lsst.multi:201 [INFO] cancelling 17096774 on provider astro proc_lsst.multi INFO: cancelling 17096774 on provider astro 2024-03-16 00:07:16 proc_lsst.multi:146 [INFO] found job 17096766 in provider astro proc_lsst.multi INFO: found job 17096766 in provider astro 2024-03-16 00:07:16 proc_lsst.multi:201 [INFO] cancelling 17096766 on provider astro proc_lsst.multi INFO: cancelling 17096766 on provider astro 2024-03-16 00:07:16 proc_lsst.multi:146 [INFO] found job 17096769 in provider astro proc_lsst.multi INFO: found job 17096769 in provider astro 2024-03-16 00:07:16 proc_lsst.multi:201 [INFO] cancelling 17096769 on provider astro proc_lsst.multi INFO: cancelling 17096769 on provider astro 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096761 in provider astro proc_lsst.multi INFO: found job 17096761 in provider astro 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096761 on provider astro proc_lsst.multi INFO: cancelling 17096761 on provider astro 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17843 in provider local proc_lsst.multi INFO: found job 17843 in provider local 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17843 on provider local proc_lsst.multi INFO: cancelling 17843 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 17843 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096780 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096780 in provider escience_rtx6k 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096780 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096780 on provider escience_rtx6k 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096782 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096782 in provider escience_rtx6k 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096782 on provider escience_rtx6k 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096790 in provider escience_a40 proc_lsst.multi INFO: found job 17096790 in provider escience_a40 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096790 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096790 on provider escience_a40 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096797 in provider escience_a40 proc_lsst.multi INFO: found job 17096797 in provider escience_a40 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096797 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096797 on provider escience_a40 2024-03-16 00:07:17 proc_lsst.multi:146 [INFO] found job 17096791 in provider escience_a40 proc_lsst.multi INFO: found job 17096791 in provider escience_a40 2024-03-16 00:07:17 proc_lsst.multi:201 [INFO] cancelling 17096791 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096791 on provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:146 [INFO] found job 17096798 in provider escience_a40 proc_lsst.multi INFO: found job 17096798 in provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:201 [INFO] cancelling 17096798 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096798 on provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:146 [INFO] found job 17096795 in provider escience_a40 proc_lsst.multi INFO: found job 17096795 in provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:201 [INFO] cancelling 17096795 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096795 on provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:146 [INFO] found job 17096796 in provider escience_a40 proc_lsst.multi INFO: found job 17096796 in provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:201 [INFO] cancelling 17096796 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096796 on provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:146 [INFO] found job 17096781 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096781 in provider escience_rtx6k 2024-03-16 00:07:18 proc_lsst.multi:201 [INFO] cancelling 17096781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096781 on provider escience_rtx6k 2024-03-16 00:07:18 proc_lsst.multi:146 [INFO] found job 17096793 in provider escience_a40 proc_lsst.multi INFO: found job 17096793 in provider escience_a40 2024-03-16 00:07:18 proc_lsst.multi:201 [INFO] cancelling 17096793 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096793 on provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096789 in provider escience_a40 proc_lsst.multi INFO: found job 17096789 in provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096789 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096789 on provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096794 in provider escience_a40 proc_lsst.multi INFO: found job 17096794 in provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096794 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096794 on provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096792 in provider escience_a40 proc_lsst.multi INFO: found job 17096792 in provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096792 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096792 on provider escience_a40 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096767 in provider astro proc_lsst.multi INFO: found job 17096767 in provider astro 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096767 on provider astro proc_lsst.multi INFO: cancelling 17096767 on provider astro 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096768 in provider astro proc_lsst.multi INFO: found job 17096768 in provider astro 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096768 on provider astro proc_lsst.multi INFO: cancelling 17096768 on provider astro 2024-03-16 00:07:19 proc_lsst.multi:146 [INFO] found job 17096776 in provider astro proc_lsst.multi INFO: found job 17096776 in provider astro 2024-03-16 00:07:19 proc_lsst.multi:201 [INFO] cancelling 17096776 on provider astro proc_lsst.multi INFO: cancelling 17096776 on provider astro 2024-03-16 00:07:20 proc_lsst.multi:146 [INFO] found job 17096779 in provider astro proc_lsst.multi INFO: found job 17096779 in provider astro 2024-03-16 00:07:20 proc_lsst.multi:201 [INFO] cancelling 17096779 on provider astro proc_lsst.multi INFO: cancelling 17096779 on provider astro 2024-03-16 00:07:20 proc_lsst.multi:146 [INFO] found job 17096778 in provider astro proc_lsst.multi INFO: found job 17096778 in provider astro 2024-03-16 00:07:20 proc_lsst.multi:201 [INFO] cancelling 17096778 on provider astro proc_lsst.multi INFO: cancelling 17096778 on provider astro 2024-03-16 00:07:20 proc_lsst.multi:146 [INFO] found job 17096777 in provider astro proc_lsst.multi INFO: found job 17096777 in provider astro 2024-03-16 00:07:20 proc_lsst.multi:201 [INFO] cancelling 17096777 on provider astro proc_lsst.multi INFO: cancelling 17096777 on provider astro 2024-03-16 00:07:20 proc_lsst.multi:146 [INFO] found job 17096758 in provider astro proc_lsst.multi INFO: found job 17096758 in provider astro 2024-03-16 00:07:20 proc_lsst.multi:201 [INFO] cancelling 17096758 on provider astro proc_lsst.multi INFO: cancelling 17096758 on provider astro 2024-03-16 00:07:21 proc_lsst.multi:146 [INFO] found job 17096751 in provider astro proc_lsst.multi INFO: found job 17096751 in provider astro 2024-03-16 00:07:21 proc_lsst.multi:201 [INFO] cancelling 17096751 on provider astro proc_lsst.multi INFO: cancelling 17096751 on provider astro 2024-03-16 00:07:21 proc_lsst.multi:146 [INFO] found job 17096752 in provider astro proc_lsst.multi INFO: found job 17096752 in provider astro 2024-03-16 00:07:21 proc_lsst.multi:201 [INFO] cancelling 17096752 on provider astro proc_lsst.multi INFO: cancelling 17096752 on provider astro 2024-03-16 00:07:21 proc_lsst.multi:146 [INFO] found job 17096755 in provider astro proc_lsst.multi INFO: found job 17096755 in provider astro 2024-03-16 00:07:21 proc_lsst.multi:201 [INFO] cancelling 17096755 on provider astro proc_lsst.multi INFO: cancelling 17096755 on provider astro 2024-03-16 00:07:21 proc_lsst.multi:146 [INFO] found job 17096754 in provider astro proc_lsst.multi INFO: found job 17096754 in provider astro 2024-03-16 00:07:21 proc_lsst.multi:201 [INFO] cancelling 17096754 on provider astro proc_lsst.multi INFO: cancelling 17096754 on provider astro 2024-03-16 00:07:21 proc_lsst.multi:146 [INFO] found job 17096753 in provider astro proc_lsst.multi INFO: found job 17096753 in provider astro 2024-03-16 00:07:21 proc_lsst.multi:201 [INFO] cancelling 17096753 on provider astro proc_lsst.multi INFO: cancelling 17096753 on provider astro 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096759 in provider astro proc_lsst.multi INFO: found job 17096759 in provider astro 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096759 on provider astro proc_lsst.multi INFO: cancelling 17096759 on provider astro 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096757 in provider astro proc_lsst.multi INFO: found job 17096757 in provider astro 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096757 on provider astro proc_lsst.multi INFO: cancelling 17096757 on provider astro 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096750 in provider astro proc_lsst.multi INFO: found job 17096750 in provider astro 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096750 on provider astro proc_lsst.multi INFO: cancelling 17096750 on provider astro 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096756 in provider astro proc_lsst.multi INFO: found job 17096756 in provider astro 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096756 on provider astro proc_lsst.multi INFO: cancelling 17096756 on provider astro 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096787 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096787 in provider escience_rtx6k 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096787 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096787 on provider escience_rtx6k 2024-03-16 00:07:22 proc_lsst.multi:146 [INFO] found job 17096788 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096788 in provider escience_rtx6k 2024-03-16 00:07:22 proc_lsst.multi:201 [INFO] cancelling 17096788 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096788 on provider escience_rtx6k 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096771 in provider astro proc_lsst.multi INFO: found job 17096771 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096771 on provider astro proc_lsst.multi INFO: cancelling 17096771 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-16 00:07:23 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-16 00:07:23 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17096750', '17096751', '17096752', '17096753', '17096754', '17096755', '17096756', '17096757', '17096758', '17096759', '17096760', '17096761', '17096762', '17096763', '17096764', '17096765', '17096766', '17096767', '17096768', '17096769', '17096770', '17096771', '17096772', '17096773', '17096774', '17096775', '17096776', '17096777', '17096778', '17096779', '17096789', '17096790', '17096791', '17096792', '17096793', '17096794', '17096795', '17096796', '17096797', '17096798', '17096780', '17096781', '17096782', '17096787', '17096788', '17843'] proc_lsst.multi INFO: new jobs since last cancel ['17096750', '17096751', '17096752', '17096753', '17096754', '17096755', '17096756', '17096757', '17096758', '17096759', '17096760', '17096761', '17096762', '17096763', '17096764', '17096765', '17096766', '17096767', '17096768', '17096769', '17096770', '17096771', '17096772', '17096773', '17096774', '17096775', '17096776', '17096777', '17096778', '17096779', '17096789', '17096790', '17096791', '17096792', '17096793', '17096794', '17096795', '17096796', '17096797', '17096798', '17096780', '17096781', '17096782', '17096787', '17096788', '17843'] 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096750 in provider astro proc_lsst.multi INFO: found job 17096750 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096750 on provider astro proc_lsst.multi INFO: cancelling 17096750 on provider astro 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096751 in provider astro proc_lsst.multi INFO: found job 17096751 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096751 on provider astro proc_lsst.multi INFO: cancelling 17096751 on provider astro 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096752 in provider astro proc_lsst.multi INFO: found job 17096752 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096752 on provider astro proc_lsst.multi INFO: cancelling 17096752 on provider astro 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096753 in provider astro proc_lsst.multi INFO: found job 17096753 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096753 on provider astro proc_lsst.multi INFO: cancelling 17096753 on provider astro 2024-03-16 00:07:23 proc_lsst.multi:146 [INFO] found job 17096754 in provider astro proc_lsst.multi INFO: found job 17096754 in provider astro 2024-03-16 00:07:23 proc_lsst.multi:201 [INFO] cancelling 17096754 on provider astro proc_lsst.multi INFO: cancelling 17096754 on provider astro 2024-03-16 00:07:24 proc_lsst.multi:146 [INFO] found job 17096755 in provider astro proc_lsst.multi INFO: found job 17096755 in provider astro 2024-03-16 00:07:24 proc_lsst.multi:201 [INFO] cancelling 17096755 on provider astro proc_lsst.multi INFO: cancelling 17096755 on provider astro 2024-03-16 00:07:24 proc_lsst.multi:146 [INFO] found job 17096756 in provider astro proc_lsst.multi INFO: found job 17096756 in provider astro 2024-03-16 00:07:24 proc_lsst.multi:201 [INFO] cancelling 17096756 on provider astro proc_lsst.multi INFO: cancelling 17096756 on provider astro 2024-03-16 00:07:24 proc_lsst.multi:146 [INFO] found job 17096757 in provider astro proc_lsst.multi INFO: found job 17096757 in provider astro 2024-03-16 00:07:24 proc_lsst.multi:201 [INFO] cancelling 17096757 on provider astro proc_lsst.multi INFO: cancelling 17096757 on provider astro 2024-03-16 00:07:24 proc_lsst.multi:146 [INFO] found job 17096758 in provider astro proc_lsst.multi INFO: found job 17096758 in provider astro 2024-03-16 00:07:24 proc_lsst.multi:201 [INFO] cancelling 17096758 on provider astro proc_lsst.multi INFO: cancelling 17096758 on provider astro 2024-03-16 00:07:24 proc_lsst.multi:146 [INFO] found job 17096759 in provider astro proc_lsst.multi INFO: found job 17096759 in provider astro 2024-03-16 00:07:24 proc_lsst.multi:201 [INFO] cancelling 17096759 on provider astro proc_lsst.multi INFO: cancelling 17096759 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096760 in provider astro proc_lsst.multi INFO: found job 17096760 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096760 on provider astro proc_lsst.multi INFO: cancelling 17096760 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096761 in provider astro proc_lsst.multi INFO: found job 17096761 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096761 on provider astro proc_lsst.multi INFO: cancelling 17096761 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096762 in provider astro proc_lsst.multi INFO: found job 17096762 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096762 on provider astro proc_lsst.multi INFO: cancelling 17096762 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096763 in provider astro proc_lsst.multi INFO: found job 17096763 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096763 on provider astro proc_lsst.multi INFO: cancelling 17096763 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096764 in provider astro proc_lsst.multi INFO: found job 17096764 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096764 on provider astro proc_lsst.multi INFO: cancelling 17096764 on provider astro 2024-03-16 00:07:25 proc_lsst.multi:146 [INFO] found job 17096765 in provider astro proc_lsst.multi INFO: found job 17096765 in provider astro 2024-03-16 00:07:25 proc_lsst.multi:201 [INFO] cancelling 17096765 on provider astro proc_lsst.multi INFO: cancelling 17096765 on provider astro 2024-03-16 00:07:26 proc_lsst.multi:146 [INFO] found job 17096766 in provider astro proc_lsst.multi INFO: found job 17096766 in provider astro 2024-03-16 00:07:26 proc_lsst.multi:201 [INFO] cancelling 17096766 on provider astro proc_lsst.multi INFO: cancelling 17096766 on provider astro 2024-03-16 00:07:26 proc_lsst.multi:146 [INFO] found job 17096767 in provider astro proc_lsst.multi INFO: found job 17096767 in provider astro 2024-03-16 00:07:26 proc_lsst.multi:201 [INFO] cancelling 17096767 on provider astro proc_lsst.multi INFO: cancelling 17096767 on provider astro 2024-03-16 00:07:26 proc_lsst.multi:146 [INFO] found job 17096768 in provider astro proc_lsst.multi INFO: found job 17096768 in provider astro 2024-03-16 00:07:26 proc_lsst.multi:201 [INFO] cancelling 17096768 on provider astro proc_lsst.multi INFO: cancelling 17096768 on provider astro 2024-03-16 00:07:26 proc_lsst.multi:146 [INFO] found job 17096769 in provider astro proc_lsst.multi INFO: found job 17096769 in provider astro 2024-03-16 00:07:26 proc_lsst.multi:201 [INFO] cancelling 17096769 on provider astro proc_lsst.multi INFO: cancelling 17096769 on provider astro 2024-03-16 00:07:26 proc_lsst.multi:146 [INFO] found job 17096770 in provider astro proc_lsst.multi INFO: found job 17096770 in provider astro 2024-03-16 00:07:26 proc_lsst.multi:201 [INFO] cancelling 17096770 on provider astro proc_lsst.multi INFO: cancelling 17096770 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096771 in provider astro proc_lsst.multi INFO: found job 17096771 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096771 on provider astro proc_lsst.multi INFO: cancelling 17096771 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096772 in provider astro proc_lsst.multi INFO: found job 17096772 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096772 on provider astro proc_lsst.multi INFO: cancelling 17096772 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096773 in provider astro proc_lsst.multi INFO: found job 17096773 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096773 on provider astro proc_lsst.multi INFO: cancelling 17096773 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096774 in provider astro proc_lsst.multi INFO: found job 17096774 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096774 on provider astro proc_lsst.multi INFO: cancelling 17096774 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096775 in provider astro proc_lsst.multi INFO: found job 17096775 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096775 on provider astro proc_lsst.multi INFO: cancelling 17096775 on provider astro 2024-03-16 00:07:27 proc_lsst.multi:146 [INFO] found job 17096776 in provider astro proc_lsst.multi INFO: found job 17096776 in provider astro 2024-03-16 00:07:27 proc_lsst.multi:201 [INFO] cancelling 17096776 on provider astro proc_lsst.multi INFO: cancelling 17096776 on provider astro 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096777 in provider astro proc_lsst.multi INFO: found job 17096777 in provider astro 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096777 on provider astro proc_lsst.multi INFO: cancelling 17096777 on provider astro 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096778 in provider astro proc_lsst.multi INFO: found job 17096778 in provider astro 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096778 on provider astro proc_lsst.multi INFO: cancelling 17096778 on provider astro 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096779 in provider astro proc_lsst.multi INFO: found job 17096779 in provider astro 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096779 on provider astro proc_lsst.multi INFO: cancelling 17096779 on provider astro 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096789 in provider escience_a40 proc_lsst.multi INFO: found job 17096789 in provider escience_a40 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096789 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096789 on provider escience_a40 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096790 in provider escience_a40 proc_lsst.multi INFO: found job 17096790 in provider escience_a40 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096790 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096790 on provider escience_a40 2024-03-16 00:07:28 proc_lsst.multi:146 [INFO] found job 17096791 in provider escience_a40 proc_lsst.multi INFO: found job 17096791 in provider escience_a40 2024-03-16 00:07:28 proc_lsst.multi:201 [INFO] cancelling 17096791 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096791 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096792 in provider escience_a40 proc_lsst.multi INFO: found job 17096792 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096792 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096792 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096793 in provider escience_a40 proc_lsst.multi INFO: found job 17096793 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096793 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096793 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096794 in provider escience_a40 proc_lsst.multi INFO: found job 17096794 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096794 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096794 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096795 in provider escience_a40 proc_lsst.multi INFO: found job 17096795 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096795 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096795 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096796 in provider escience_a40 proc_lsst.multi INFO: found job 17096796 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096796 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096796 on provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:146 [INFO] found job 17096797 in provider escience_a40 proc_lsst.multi INFO: found job 17096797 in provider escience_a40 2024-03-16 00:07:29 proc_lsst.multi:201 [INFO] cancelling 17096797 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096797 on provider escience_a40 2024-03-16 00:07:30 proc_lsst.multi:146 [INFO] found job 17096798 in provider escience_a40 proc_lsst.multi INFO: found job 17096798 in provider escience_a40 2024-03-16 00:07:30 proc_lsst.multi:201 [INFO] cancelling 17096798 on provider escience_a40 proc_lsst.multi INFO: cancelling 17096798 on provider escience_a40 2024-03-16 00:07:30 proc_lsst.multi:146 [INFO] found job 17096780 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096780 in provider escience_rtx6k 2024-03-16 00:07:30 proc_lsst.multi:201 [INFO] cancelling 17096780 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096780 on provider escience_rtx6k 2024-03-16 00:07:30 proc_lsst.multi:146 [INFO] found job 17096781 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096781 in provider escience_rtx6k 2024-03-16 00:07:30 proc_lsst.multi:201 [INFO] cancelling 17096781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096781 on provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:146 [INFO] found job 17096782 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096782 in provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:201 [INFO] cancelling 17096782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096782 on provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:146 [INFO] found job 17096787 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096787 in provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:201 [INFO] cancelling 17096787 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096787 on provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:146 [INFO] found job 17096788 in provider escience_rtx6k proc_lsst.multi INFO: found job 17096788 in provider escience_rtx6k 2024-03-16 00:07:31 proc_lsst.multi:201 [INFO] cancelling 17096788 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17096788 on provider escience_rtx6k 2024-03-16 00:07:32 proc_lsst.multi:146 [INFO] found job 17843 in provider local proc_lsst.multi INFO: found job 17843 in provider local 2024-03-16 00:07:32 proc_lsst.multi:201 [INFO] cancelling 17843 on provider local proc_lsst.multi INFO: cancelling 17843 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 17843 parsl.providers.local.local WARNING: Failed to kill PID: 17843 and child processes on local 2024-03-16 00:07:33 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 106.8798 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210518_A1f_science#step0_20240316T000537Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 106.8859 seconds; current memory usage: 0.272 Gibyte, delta: -0.002 Gibyte, peak delta: 0.004 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 116.9631 seconds; current memory usage: 0.272 Gibyte, delta: 0.135 Gibyte, peak delta: 0.141 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.278 Gibyte (main), 0.277 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210518_A1f_science#step0_20240316T000537Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 1m58.843s user 0m30.687s sys 0m14.138s