+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20210908/B1c --output-run DEEP/20210908/B1c/science#step0/20240317T012544Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 11.6306 seconds; current memory usage: 0.180 Gibyte, delta: 0.041 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0025 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0079 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/pipeline_orig.qgraph DEEP/20210908/B1c/science#step0/20240317T012544Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.5466 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 6.6459 seconds; current memory usage: 0.272 Gibyte, delta: 0.092 Gibyte, peak delta: 0.080 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0099 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 186 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20210908_B1c_science#step0_20240317T012544Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0482 seconds; current memory usage: 0.272 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.272 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 188 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2214 seconds; current memory usage: 0.273 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.273 Gibyte (main), 0.261 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: ee3ce5ef-d444-4b9a-ac89-82125c20fe43 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-17 01:26:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-17 01:26:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 01:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-17 01:26:17 proc_lsst.multi:170 [INFO] [multi] job_id 53465 proc_lsst.multi INFO: [multi] job_id 53465 2024-03-17 01:26:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:17 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 53465 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 53465 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 53465 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 62454ff9da77a39e6e034c383cc726cc parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4131bd7f-78b6-43f9-bd96-eef40eb21027_isrForCrosstalkSources_1031141_7.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4131bd7f-78b6-43f9-bd96-eef40eb21027_isrForCrosstalkSources_1031141_7.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 6de594ad26492f5b22081fe3550680b0 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/c2b542e9-8aa1-4edb-88a2-138f67b82cfa_isrForCrosstalkSources_1031139_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/c2b542e9-8aa1-4edb-88a2-138f67b82cfa_isrForCrosstalkSources_1031139_52.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash 03d09b674def21d7d1229937c6447b32 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/19e9311d-923e-403e-8f1e-c2b9b371120c_isrForCrosstalkSources_1031141_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/19e9311d-923e-403e-8f1e-c2b9b371120c_isrForCrosstalkSources_1031141_51.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 721961918518914f88079382851800ef parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/88683b2b-a5bc-416c-9aed-55f8980ef482_isrForCrosstalkSources_1031140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/88683b2b-a5bc-416c-9aed-55f8980ef482_isrForCrosstalkSources_1031140_30.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash b9f24b23f9260a03f3d8f82555a4702b parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9623c7c4-6505-4f2b-a148-398263ebf086_isrForCrosstalkSources_1031140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9623c7c4-6505-4f2b-a148-398263ebf086_isrForCrosstalkSources_1031140_27.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash c122fe9dcd5a54c49ef66801d430cc6e parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/be869998-97bf-4931-b86b-70d912e6de51_isrForCrosstalkSources_1031139_35.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/be869998-97bf-4931-b86b-70d912e6de51_isrForCrosstalkSources_1031139_35.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash 69dc938499b9212d340a3c816b8f266f parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/44766ab3-4011-4052-b7a2-a2db97b95568_isrForCrosstalkSources_1031140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/44766ab3-4011-4052-b7a2-a2db97b95568_isrForCrosstalkSources_1031140_3.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash de20627cb3262dcd156c8fe5b8212489 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18ee3a5b-3583-495a-b726-b3c11b0c7536_isrForCrosstalkSources_1031141_36.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18ee3a5b-3583-495a-b726-b3c11b0c7536_isrForCrosstalkSources_1031141_36.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash 19596a64d530be040f627b40f7b78bdb parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18321e31-cc38-43b4-88e3-d1e8fbb24269_isrForCrosstalkSources_1031141_37.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18321e31-cc38-43b4-88e3-d1e8fbb24269_isrForCrosstalkSources_1031141_37.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 0cdcb17161b95d457afe6d150d453494 parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ed15aa3f-4871-4e85-880e-1eb94c03c1ad_isrForCrosstalkSources_1031141_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ed15aa3f-4871-4e85-880e-1eb94c03c1ad_isrForCrosstalkSources_1031141_5.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 781f6b6e0964d3f77b7811ab0dc0191b parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c2c99be4-d060-440f-a043-4964e48a2367_isrForCrosstalkSources_1031141_39.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c2c99be4-d060-440f-a043-4964e48a2367_isrForCrosstalkSources_1031141_39.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 81a9c06d4717c5258a5e63239144565c parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2bfbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/535fb1c2-815b-41d1-a0d3-959d2b5d3b75_isrForCrosstalkSources_1031140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/535fb1c2-815b-41d1-a0d3-959d2b5d3b75_isrForCrosstalkSources_1031140_13.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash caa0fa66cfe84e13f00b2ca0d2d6c9ea parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8abc552d-f01b-4cb4-831f-1eddba502851_isrForCrosstalkSources_1031141_12.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8abc552d-f01b-4cb4-831f-1eddba502851_isrForCrosstalkSources_1031141_12.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash a1dcc72c18add70d142a67bf0f000695 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/91212434-6211-4358-ad0c-5fc6ecccab30_isrForCrosstalkSources_1031140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/91212434-6211-4358-ad0c-5fc6ecccab30_isrForCrosstalkSources_1031140_62.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash e10b5687332d76b0bf84561c97d406ea parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7db773be-a774-4c28-a384-d26a23d130b7_isrForCrosstalkSources_1031139_6.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7db773be-a774-4c28-a384-d26a23d130b7_isrForCrosstalkSources_1031139_6.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash b55a6dc2134c60f39df99ceb6d40b68c parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/63f808a9-fcf8-4f05-a863-43c52b14e9d5_isrForCrosstalkSources_1031141_48.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/63f808a9-fcf8-4f05-a863-43c52b14e9d5_isrForCrosstalkSources_1031141_48.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 55d6d501ec93a3d467a34103dc8e34a0 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/55980193-cf50-4a29-9093-95f788e7ccc5_isrForCrosstalkSources_1031139_33.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/55980193-cf50-4a29-9093-95f788e7ccc5_isrForCrosstalkSources_1031139_33.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash f6bbd1a663daa91c58542186c5b3b3a7 parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/ccaabdb5-4488-4f61-a960-fb03815e62ea_isrForCrosstalkSources_1031139_48.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/ccaabdb5-4488-4f61-a960-fb03815e62ea_isrForCrosstalkSources_1031139_48.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 12293e80796796ec9a62e45dfb5ab7c3 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8a7e7a-c0c5-44df-bbf6-6ce86a71f506_isrForCrosstalkSources_1031139_24.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8a7e7a-c0c5-44df-bbf6-6ce86a71f506_isrForCrosstalkSources_1031139_24.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 595fdef3ae55be533a715cae272ce5e9 parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2e291ab2-98db-4b85-94ab-3fd04a5613f7_isrForCrosstalkSources_1031140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2e291ab2-98db-4b85-94ab-3fd04a5613f7_isrForCrosstalkSources_1031140_51.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash d62f2b7dcb420d9c328046d16bbdf114 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/821ebf1a-e7e3-4863-8a93-0f4bf10010f4_isrForCrosstalkSources_1031141_3.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/821ebf1a-e7e3-4863-8a93-0f4bf10010f4_isrForCrosstalkSources_1031141_3.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash b060f325ecb3cc1d90d457464c3d71c6 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1bc6bf0d-999e-413e-bf9e-bc9d4f977238_isrForCrosstalkSources_1031141_2.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1bc6bf0d-999e-413e-bf9e-bc9d4f977238_isrForCrosstalkSources_1031141_2.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 0bae3575465f5634979349e07478897f parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4ebcb221-b3e3-404a-85a6-91ec0b33a3cf_isrForCrosstalkSources_1031139_21.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4ebcb221-b3e3-404a-85a6-91ec0b33a3cf_isrForCrosstalkSources_1031139_21.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash 05636602d29c700e085324a186d80fec parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7f2c9697-eff7-4fd5-8b4e-ff7c9a19296c_isrForCrosstalkSources_1031139_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7f2c9697-eff7-4fd5-8b4e-ff7c9a19296c_isrForCrosstalkSources_1031139_4.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash cd2b78d16c2906cf6146fae64238492d parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a142e785-3642-4d04-9836-72a679d8b7f0_isrForCrosstalkSources_1031139_26.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a142e785-3642-4d04-9836-72a679d8b7f0_isrForCrosstalkSources_1031139_26.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 881f4dbcf5ca042237a288c411ace957 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3d3343df-6edf-4921-ab0a-7654357d5921_isrForCrosstalkSources_1031139_10.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3d3343df-6edf-4921-ab0a-7654357d5921_isrForCrosstalkSources_1031139_10.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash 57b686bd0877e1edc4ddd274c3bee328 parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/86635514-ec59-4a5e-bc56-e3cb4f11152e_isrForCrosstalkSources_1031140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/86635514-ec59-4a5e-bc56-e3cb4f11152e_isrForCrosstalkSources_1031140_20.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 743e7caba99f5a0b326a88b1fd02363d parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/292d0d9c-99a2-4bc2-bd4e-22dcfc855429_isrForCrosstalkSources_1031141_33.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/292d0d9c-99a2-4bc2-bd4e-22dcfc855429_isrForCrosstalkSources_1031141_33.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash 258c62db79699572cc6d682862d08d6a parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/9098554f-88a6-4309-869d-7d82e5650dda_isrForCrosstalkSources_1031139_43.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/9098554f-88a6-4309-869d-7d82e5650dda_isrForCrosstalkSources_1031139_43.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 5178ed81ec4006ab345cd0b2c8476925 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/978110b6-8dca-444b-a506-ca8e24e09bcd_isrForCrosstalkSources_1031140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/978110b6-8dca-444b-a506-ca8e24e09bcd_isrForCrosstalkSources_1031140_6.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 67750d22ed9d6d8458382a09cdd12c2d parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/645fe08d-3297-483e-b3f3-2a74fa4b4843_isrForCrosstalkSources_1031141_6.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/645fe08d-3297-483e-b3f3-2a74fa4b4843_isrForCrosstalkSources_1031141_6.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 0ac77a177fd05cb37d9a6772121b9c7f parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b229ab16-45fa-460c-8288-6d093ecce3c2_isrForCrosstalkSources_1031141_15.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b229ab16-45fa-460c-8288-6d093ecce3c2_isrForCrosstalkSources_1031141_15.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash 6a86e53d6fcc6d382dbc23c0f6aa5a30 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b65f50d3-3a6a-41bc-a4b0-da584f0a883f_isrForCrosstalkSources_1031140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b65f50d3-3a6a-41bc-a4b0-da584f0a883f_isrForCrosstalkSources_1031140_16.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 65264c6630b29815b359ed99831fd62b parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fc9e25f1-a459-4eb4-9a05-110412ea086f_isrForCrosstalkSources_1031140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fc9e25f1-a459-4eb4-9a05-110412ea086f_isrForCrosstalkSources_1031140_19.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 41f8f013937f0bba00d4ad2345c2f687 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fdfb1bea-7a7f-4a72-bbf1-87c177b82a80_isrForCrosstalkSources_1031139_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fdfb1bea-7a7f-4a72-bbf1-87c177b82a80_isrForCrosstalkSources_1031139_16.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 5e1ce25357769c2a32a2504c7a6c9942 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/12177999-83e5-4dac-a759-f97fcb41bcf8_isrForCrosstalkSources_1031141_16.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/12177999-83e5-4dac-a759-f97fcb41bcf8_isrForCrosstalkSources_1031141_16.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash ff1a5e9eb7c180b51389b2c5d9b7de57 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0c005b82-1bf5-4c20-accc-d75c5f14755a_isrForCrosstalkSources_1031139_46.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0c005b82-1bf5-4c20-accc-d75c5f14755a_isrForCrosstalkSources_1031139_46.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash baf060c055d34bba0befdf6ad0b48e43 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ddca9636-042c-4e30-8e4e-59e9a57472a8_isrForCrosstalkSources_1031141_10.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ddca9636-042c-4e30-8e4e-59e9a57472a8_isrForCrosstalkSources_1031141_10.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 1f3aa4e46422d1091d3e6d13f8dc5493 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/64acc5b2-7a1a-4578-a200-0cd5222c2e84_isrForCrosstalkSources_1031141_55.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/64acc5b2-7a1a-4578-a200-0cd5222c2e84_isrForCrosstalkSources_1031141_55.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 8b46cb030b10f7502ce190edd897b805 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/777d4ca4-fcd8-46a7-9cd4-e35e64771508_isrForCrosstalkSources_1031140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/777d4ca4-fcd8-46a7-9cd4-e35e64771508_isrForCrosstalkSources_1031140_57.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash f8773183d29d5bed59edb4f68da87a67 parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f9587144-2a60-48ea-9019-bb66f72896d0_isrForCrosstalkSources_1031141_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f9587144-2a60-48ea-9019-bb66f72896d0_isrForCrosstalkSources_1031141_32.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 2708e198fa982e1e38c165e1ebca0c3b parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6dfbd6cf-8a7c-44d7-8a5f-893c0db0af92_isrForCrosstalkSources_1031139_1.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6dfbd6cf-8a7c-44d7-8a5f-893c0db0af92_isrForCrosstalkSources_1031139_1.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash c24a485cf2d7c7740655c1b81c7b6e81 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bcbf4c3d-463f-48af-bd0b-b86da68182be_isrForCrosstalkSources_1031140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bcbf4c3d-463f-48af-bd0b-b86da68182be_isrForCrosstalkSources_1031140_14.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash cc586df296472e20a393b588563494fb parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/97d0bc63-00fb-43a7-8622-921a1f94d70c_isrForCrosstalkSources_1031139_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/97d0bc63-00fb-43a7-8622-921a1f94d70c_isrForCrosstalkSources_1031139_44.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 86890fcc0e3f4a5e1722c50994ddd920 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/3a5382b9-2a89-45ba-bf07-99faf834fc8e_isrForCrosstalkSources_1031140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/3a5382b9-2a89-45ba-bf07-99faf834fc8e_isrForCrosstalkSources_1031140_44.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash 51baa358513eed010c66d49b77cc67ea parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/11ad3842-4314-46ee-8652-efc4381814fb_isrForCrosstalkSources_1031139_53.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/11ad3842-4314-46ee-8652-efc4381814fb_isrForCrosstalkSources_1031139_53.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 4cdeb498008004d7993cf6dc7dc4920f parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/547c3dda-6383-452a-98fc-8707398e7f0b_isrForCrosstalkSources_1031139_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/547c3dda-6383-452a-98fc-8707398e7f0b_isrForCrosstalkSources_1031139_15.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 5031d9365589d04ed2a4c06d3e7315a4 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/91de433f-6efc-4616-b537-eb51789bd39a_isrForCrosstalkSources_1031141_46.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/91de433f-6efc-4616-b537-eb51789bd39a_isrForCrosstalkSources_1031141_46.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash d978723d1ac45d6d2bcf3eef34e25745 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fb3227df-186b-4185-8af1-22b3eb201a0d_isrForCrosstalkSources_1031139_37.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fb3227df-186b-4185-8af1-22b3eb201a0d_isrForCrosstalkSources_1031139_37.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash 155ee2e3d19154bdeb659cecb850bb5d parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f3ab147e-0dd2-454c-b011-5c20796557d8_isrForCrosstalkSources_1031140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f3ab147e-0dd2-454c-b011-5c20796557d8_isrForCrosstalkSources_1031140_12.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash ba5805c17b2ddf6cb4bf2d07b2e5f881 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d386a25c-af89-467b-a7d8-924b1ba4e66c_isrForCrosstalkSources_1031139_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d386a25c-af89-467b-a7d8-924b1ba4e66c_isrForCrosstalkSources_1031139_27.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash c8a6a185618f4f68eef9c1131741b183 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3dc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e07c859a-d2fb-4353-a5a9-0472ccf365b4_isrForCrosstalkSources_1031140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e07c859a-d2fb-4353-a5a9-0472ccf365b4_isrForCrosstalkSources_1031140_36.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 4b6bd9cdea8d796b14245d1e601e99e1 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3ba8d41c-c3c3-4ef6-972b-04e04d2f20ca_isrForCrosstalkSources_1031139_29.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3ba8d41c-c3c3-4ef6-972b-04e04d2f20ca_isrForCrosstalkSources_1031139_29.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 6d7d3e9dae0bf494ab93f1cd871003d8 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6ed4700e-18e7-4523-9415-bb8de1a6c156_isrForCrosstalkSources_1031139_12.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6ed4700e-18e7-4523-9415-bb8de1a6c156_isrForCrosstalkSources_1031139_12.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash a1c6d57d5b9f5d97cda0a4865bd81325 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/0ac8a91a-8d99-4b3d-ba82-70bc5b9d72a0_isrForCrosstalkSources_1031140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/0ac8a91a-8d99-4b3d-ba82-70bc5b9d72a0_isrForCrosstalkSources_1031140_46.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 7ce08c2f49eb78b72b21ada43ddb3bdc parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8540ccb9-0865-4ab2-a75c-31e1d35668e6_isrForCrosstalkSources_1031139_22.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8540ccb9-0865-4ab2-a75c-31e1d35668e6_isrForCrosstalkSources_1031139_22.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash db9b36f70374020972e566410a8ef422 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9ea4f61f-a3a6-47e4-a28f-2011ae5a5552_isrForCrosstalkSources_1031140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9ea4f61f-a3a6-47e4-a28f-2011ae5a5552_isrForCrosstalkSources_1031140_9.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash 893a667290ac67b9685865e080d39a7e parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8a03fa65-740a-41bc-a27d-825f2d34b497_isrForCrosstalkSources_1031139_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8a03fa65-740a-41bc-a27d-825f2d34b497_isrForCrosstalkSources_1031139_45.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash a3023d72762fbe597a2f441c48d6845b parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1ff4f013-5655-4e6c-8d0d-35cd5c721f37_isrForCrosstalkSources_1031140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1ff4f013-5655-4e6c-8d0d-35cd5c721f37_isrForCrosstalkSources_1031140_32.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 360362025e6f88739b1c2d63654189e4 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1c4905dd-ece5-4433-b58e-6b1fef2ed503_isrForCrosstalkSources_1031141_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1c4905dd-ece5-4433-b58e-6b1fef2ed503_isrForCrosstalkSources_1031141_24.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 4326239a675cced532a599e2f852db40 parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49b7dfb3-09de-443a-8ec2-54e04297f2c4_isrForCrosstalkSources_1031139_30.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49b7dfb3-09de-443a-8ec2-54e04297f2c4_isrForCrosstalkSources_1031139_30.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash ca2a0b118869e0e8bb38ffed6bf689e4 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b67671b5-82d1-4ff1-ab3b-c973ba34d4f2_isrForCrosstalkSources_1031140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b67671b5-82d1-4ff1-ab3b-c973ba34d4f2_isrForCrosstalkSources_1031140_33.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 5371705f565198e3a529babf4f2033b3 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/95717a0d-1523-4a7e-8642-b01669811496_isrForCrosstalkSources_1031139_11.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/95717a0d-1523-4a7e-8642-b01669811496_isrForCrosstalkSources_1031139_11.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash c629db5398696c53885a3ee134e92ed8 parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ee51e2fc-813f-404a-bd2f-118fe7542826_isrForCrosstalkSources_1031141_25.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ee51e2fc-813f-404a-bd2f-118fe7542826_isrForCrosstalkSources_1031141_25.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 2463b65e452449c2736e23e0be8eae2b parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5b602c36-a7dc-4d3e-ae06-ac850aa978ac_isrForCrosstalkSources_1031140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5b602c36-a7dc-4d3e-ae06-ac850aa978ac_isrForCrosstalkSources_1031140_48.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash ffb7a7722f84f6fcb39876ccab19486b parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/2ff2d47a-10e1-49af-88e5-25f8ecbfa131_isrForCrosstalkSources_1031139_36.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/2ff2d47a-10e1-49af-88e5-25f8ecbfa131_isrForCrosstalkSources_1031139_36.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 133710fd2d36914024e286fbbba0209d parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/108ac57c-8cf5-4002-a450-b930e00a52a1_isrForCrosstalkSources_1031141_52.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/108ac57c-8cf5-4002-a450-b930e00a52a1_isrForCrosstalkSources_1031141_52.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash b93952c333f42cd3471dacdd7f94e034 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/7ed409fd-0d32-495d-8276-3c2215d2c3b0_isrForCrosstalkSources_1031140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/7ed409fd-0d32-495d-8276-3c2215d2c3b0_isrForCrosstalkSources_1031140_26.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash dcb68b40f192064c4cc52f5d0677b2d6 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/73c2c216-063a-48ff-a6b2-29cb09e882a5_isrForCrosstalkSources_1031139_38.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/73c2c216-063a-48ff-a6b2-29cb09e882a5_isrForCrosstalkSources_1031139_38.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash b8b20cb2f4065a6890ed8267cdbf3b2a parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/23293ee9-6c24-4a9d-91a4-d0a3e026bdcb_isrForCrosstalkSources_1031139_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/23293ee9-6c24-4a9d-91a4-d0a3e026bdcb_isrForCrosstalkSources_1031139_42.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash b18fe8d668142f2d84851664abe5b8a9 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f807edda-e310-48d9-82c9-40ae011875b8_isrForCrosstalkSources_1031141_49.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f807edda-e310-48d9-82c9-40ae011875b8_isrForCrosstalkSources_1031141_49.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash b05a72341357aeae951ce0e29cc1db03 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/735db864-6318-4ed9-bf96-1b34c6b1f9de_isrForCrosstalkSources_1031140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/735db864-6318-4ed9-bf96-1b34c6b1f9de_isrForCrosstalkSources_1031140_4.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash 04f5aa34534fb7102c1e994f83a03931 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5fff2fd1-6936-4be6-a98d-5443495d697d_isrForCrosstalkSources_1031141_47.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5fff2fd1-6936-4be6-a98d-5443495d697d_isrForCrosstalkSources_1031141_47.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash 66d04356283d04bc0fb11d488a5a5174 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0bdf451e-68ee-4cb8-80c2-1c7a5569e506_isrForCrosstalkSources_1031141_57.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0bdf451e-68ee-4cb8-80c2-1c7a5569e506_isrForCrosstalkSources_1031141_57.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash 09e5bb3d3cdb05b5dfd623933f6f867c parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6d779be7-93f0-43e3-b888-5159c3232f93_isrForCrosstalkSources_1031140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6d779be7-93f0-43e3-b888-5159c3232f93_isrForCrosstalkSources_1031140_53.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash a3a2465a78c6dac1c079a1f348caad37 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b2dc2415-1d4b-4097-8594-9b762c2501e6_isrForCrosstalkSources_1031139_3.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b2dc2415-1d4b-4097-8594-9b762c2501e6_isrForCrosstalkSources_1031139_3.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash e19b68e9d4bfa554d76762bf1bfeb257 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/01575a4a-1a42-469d-b490-e1d84cd591e3_isrForCrosstalkSources_1031139_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/01575a4a-1a42-469d-b490-e1d84cd591e3_isrForCrosstalkSources_1031139_7.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 416ff3968d17f20854380aa1ca60169f parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1d35a7bc-9b04-4a68-9657-e50ff39bf387_isrForCrosstalkSources_1031140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1d35a7bc-9b04-4a68-9657-e50ff39bf387_isrForCrosstalkSources_1031140_23.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 80497456bfa3d0f07138df17b52de8cf parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4960920d-6960-48df-9be2-4e2c20daf1c6_isrForCrosstalkSources_1031139_50.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4960920d-6960-48df-9be2-4e2c20daf1c6_isrForCrosstalkSources_1031139_50.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 5bd1c1fa4b67c75c03bc17b796c18381 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/962b8357-6872-46c3-9e83-6f1f4d652e94_isrForCrosstalkSources_1031140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/962b8357-6872-46c3-9e83-6f1f4d652e94_isrForCrosstalkSources_1031140_45.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 13e4da44a644adc02489abebac980482 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/35bc6bf7-6368-403b-a001-633a09c93cec_isrForCrosstalkSources_1031139_20.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/35bc6bf7-6368-403b-a001-633a09c93cec_isrForCrosstalkSources_1031139_20.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash c8d7ac4fd5b35aea4782180a0ab7f093 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/459745a3-1be8-4f17-82f8-2a23d761bc0b_isrForCrosstalkSources_1031140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/459745a3-1be8-4f17-82f8-2a23d761bc0b_isrForCrosstalkSources_1031140_11.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 0d89336cbf0c46cd0820c54194d2d75e parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/163b163a-a213-407a-87c4-26e25d189301_isrForCrosstalkSources_1031140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/163b163a-a213-407a-87c4-26e25d189301_isrForCrosstalkSources_1031140_49.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash f55a3e1f0f0f45485f794c12c2f9912f parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/973f6170-553c-47ad-88fd-371d0067294a_isrForCrosstalkSources_1031141_13.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/973f6170-553c-47ad-88fd-371d0067294a_isrForCrosstalkSources_1031141_13.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 2ebda810009fd788998bfb240087a562 parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d5258297-c0a8-4aad-9c86-16ea9cbcef3d_isrForCrosstalkSources_1031139_51.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d5258297-c0a8-4aad-9c86-16ea9cbcef3d_isrForCrosstalkSources_1031139_51.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash 7c5d04218b5931209a8fcd7fe49eea31 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/85ba50be-b139-481b-bdb9-31041f8765e6_isrForCrosstalkSources_1031140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/85ba50be-b139-481b-bdb9-31041f8765e6_isrForCrosstalkSources_1031140_61.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash b9f2634f54f96e9a62b76b5a57417ecc parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/63068b92-647c-4eb3-be56-e0d816a19732_isrForCrosstalkSources_1031140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/63068b92-647c-4eb3-be56-e0d816a19732_isrForCrosstalkSources_1031140_40.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash f1fdb9acb20d717535dde157496e0394 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/be30e73d-8c30-4be6-908b-059d46eea585_isrForCrosstalkSources_1031141_1.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/be30e73d-8c30-4be6-908b-059d46eea585_isrForCrosstalkSources_1031141_1.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 569f7708ea3bf15339fe1a2f209fc03b parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5caec8f7-ee68-4689-aeec-a83b0fe04688_isrForCrosstalkSources_1031141_40.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5caec8f7-ee68-4689-aeec-a83b0fe04688_isrForCrosstalkSources_1031141_40.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 9855d12c0592177474a761e770be41c6 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6be435ee-8066-4c1b-9cf9-8c35e036d5f5_isrForCrosstalkSources_1031139_31.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6be435ee-8066-4c1b-9cf9-8c35e036d5f5_isrForCrosstalkSources_1031139_31.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash f887fcda17177f82a80cab8bf4e59f17 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61ff1abb-136e-44b1-91e3-48553c96ad49_isrForCrosstalkSources_1031140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61ff1abb-136e-44b1-91e3-48553c96ad49_isrForCrosstalkSources_1031140_34.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash d96acb820cd503645a31d98846211c41 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d3d05bb5-69c8-48d4-abab-1e81862a9420_isrForCrosstalkSources_1031139_56.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d3d05bb5-69c8-48d4-abab-1e81862a9420_isrForCrosstalkSources_1031139_56.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 64d94b598ab9ba392806c97bbc0ef7b3 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7391efe6-19f8-4f66-84d1-e3d8808c71aa_isrForCrosstalkSources_1031141_17.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7391efe6-19f8-4f66-84d1-e3d8808c71aa_isrForCrosstalkSources_1031141_17.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 4c353f86e8fc98a52129c461fbf3d348 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/9e6a074b-933f-41d2-8810-d4563bc0e841_isrForCrosstalkSources_1031141_62.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/9e6a074b-933f-41d2-8810-d4563bc0e841_isrForCrosstalkSources_1031141_62.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash ee38206f5578ab7946d1a122bd118e5e parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/bd63a19c-d087-4d30-acb4-d685e3561949_isrForCrosstalkSources_1031139_40.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/bd63a19c-d087-4d30-acb4-d685e3561949_isrForCrosstalkSources_1031139_40.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash ac91d8d0e9b83c2abd8bb609384327c3 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/088ca952-139a-41b4-a811-98fffc32ed57_isrForCrosstalkSources_1031139_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/088ca952-139a-41b4-a811-98fffc32ed57_isrForCrosstalkSources_1031139_2.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 3b87160d6404576a1421d167d00310d3 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/99eb7a64-db07-41e2-8cfe-5f4c749dc1c1_isrForCrosstalkSources_1031141_26.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/99eb7a64-db07-41e2-8cfe-5f4c749dc1c1_isrForCrosstalkSources_1031141_26.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 311e5276bec1666a5c243c0eba9e7296 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bdd6906f-e50a-460f-82f9-495c4c7af655_isrForCrosstalkSources_1031140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bdd6906f-e50a-460f-82f9-495c4c7af655_isrForCrosstalkSources_1031140_25.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash 8efc5578c49fb88554634def7d283e2f parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e2e6e8f0-e547-4f51-8abd-42c45d41a03c_isrForCrosstalkSources_1031141_20.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e2e6e8f0-e547-4f51-8abd-42c45d41a03c_isrForCrosstalkSources_1031141_20.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 77ca0a48a79f0da421fc08d4d420e966 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/16b927a1-b2bd-4a99-be2c-07e0c06ec22c_isrForCrosstalkSources_1031140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/16b927a1-b2bd-4a99-be2c-07e0c06ec22c_isrForCrosstalkSources_1031140_39.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 2faa4a5d58f8da9a859d49654157005a parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b81f6d17-3a04-46b5-a3af-64f519f47938_isrForCrosstalkSources_1031141_4.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b81f6d17-3a04-46b5-a3af-64f519f47938_isrForCrosstalkSources_1031141_4.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash ce13ff131721be6d83ea33598b6a4c63 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/43391194-1001-4b59-b548-3ef473d97837_isrForCrosstalkSources_1031139_55.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/43391194-1001-4b59-b548-3ef473d97837_isrForCrosstalkSources_1031139_55.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 7222e68c39104efb57275c7f8ff38385 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/89c7ea18-7040-4d11-b9ba-e992dd58c469_isrForCrosstalkSources_1031139_28.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/89c7ea18-7040-4d11-b9ba-e992dd58c469_isrForCrosstalkSources_1031139_28.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 410d2118c89df913fd43ab7a2a1fc7ad parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e9ab3015-ceaa-4371-b36e-44c84d77c352_isrForCrosstalkSources_1031141_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e9ab3015-ceaa-4371-b36e-44c84d77c352_isrForCrosstalkSources_1031141_44.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 1b8a416147c10ae50a28bcfb97da0a8f parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/f235b0f3-cb7a-4a39-8170-3219da353b14_isrForCrosstalkSources_1031139_8.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/f235b0f3-cb7a-4a39-8170-3219da353b14_isrForCrosstalkSources_1031139_8.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 7cd5751ac4583f429a6fa2261d678990 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b43bc7a1-f41c-4f95-9bfa-fc9f1d255649_isrForCrosstalkSources_1031141_11.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b43bc7a1-f41c-4f95-9bfa-fc9f1d255649_isrForCrosstalkSources_1031141_11.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 30e9a4ecfb911c08b664ae0e101da17d parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/fc899088-9444-43f4-be2d-bfb3640f993f_isrForCrosstalkSources_1031141_28.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/fc899088-9444-43f4-be2d-bfb3640f993f_isrForCrosstalkSources_1031141_28.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash 7728a6857460abfc000f462e417ea3a4 parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/30ef094e-3743-4f93-9ba4-461d830abc7e_isrForCrosstalkSources_1031139_54.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/30ef094e-3743-4f93-9ba4-461d830abc7e_isrForCrosstalkSources_1031139_54.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash ce5a167313862ce9c523f3f75b6d808a parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a2c3c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/3819be19-b762-4fb1-a8d2-092a59bb958e_isrForCrosstalkSources_1031141_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/3819be19-b762-4fb1-a8d2-092a59bb958e_isrForCrosstalkSources_1031141_42.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash dee25e84761a9bd450c564a971a02052 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16940e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/83db9937-2b16-4ac1-b409-5de6c47e4d00_isrForCrosstalkSources_1031139_14.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/83db9937-2b16-4ac1-b409-5de6c47e4d00_isrForCrosstalkSources_1031139_14.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash c2fc9098cae190b3050e6dd461587dc5 parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/30a4cbf4-3aab-4ba4-92fc-91aca718d001_isrForCrosstalkSources_1031140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/30a4cbf4-3aab-4ba4-92fc-91aca718d001_isrForCrosstalkSources_1031140_50.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash 81daaa6c2eee3f457e813b10a0e749f9 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/118c8a93-1097-4551-a897-5a33e8fb08a2_isrForCrosstalkSources_1031139_39.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/118c8a93-1097-4551-a897-5a33e8fb08a2_isrForCrosstalkSources_1031139_39.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash e00fa37582bff54485d988c405da2c55 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16942c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/52585796-127a-413a-9a0e-523bcf34234a_isrForCrosstalkSources_1031140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/52585796-127a-413a-9a0e-523bcf34234a_isrForCrosstalkSources_1031140_41.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 3a188d785172eb27f82cef3c44c373f5 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4d633103-52ad-4649-ab44-5268babfe8d3_isrForCrosstalkSources_1031141_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4d633103-52ad-4649-ab44-5268babfe8d3_isrForCrosstalkSources_1031141_22.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 85b382f02c61a5cb3583df72983702d1 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eb7311c-b830-4a89-8cbd-283587fe6eca_isrForCrosstalkSources_1031140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eb7311c-b830-4a89-8cbd-283587fe6eca_isrForCrosstalkSources_1031140_58.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 285de97bd2face2fff0884609406892f parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/72db7b6d-5ba9-4546-9099-a1258147f8ea_isrForCrosstalkSources_1031139_34.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/72db7b6d-5ba9-4546-9099-a1258147f8ea_isrForCrosstalkSources_1031139_34.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash 6072c44a0972138491b9dfc22319e2e5 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16944a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8de2a4b1-2bf5-43ba-9f96-7261bb592b49_isrForCrosstalkSources_1031139_49.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8de2a4b1-2bf5-43ba-9f96-7261bb592b49_isrForCrosstalkSources_1031139_49.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 91ef060d27554f11e34e627557b44ba5 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16945e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/dd1eb700-af67-424d-8457-12c93c06f6a0_isrForCrosstalkSources_1031140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/dd1eb700-af67-424d-8457-12c93c06f6a0_isrForCrosstalkSources_1031140_56.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash d628b64473bd62bc92271828b3da90ca parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/14f2c932-c2af-4116-be04-c2421dc7948a_isrForCrosstalkSources_1031139_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/14f2c932-c2af-4116-be04-c2421dc7948a_isrForCrosstalkSources_1031139_57.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 2d8e4cd4bf40a83dec6838a880a389ff parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/411fdfe4-c8a6-415e-8259-437740463842_isrForCrosstalkSources_1031140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/411fdfe4-c8a6-415e-8259-437740463842_isrForCrosstalkSources_1031140_15.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 2fbc17f44fdd8d6166ffa76157e189e0 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6cd6ab74-ce11-434b-a975-08a8cbc182d1_isrForCrosstalkSources_1031140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6cd6ab74-ce11-434b-a975-08a8cbc182d1_isrForCrosstalkSources_1031140_60.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash a2d0628a07966fd54dd6d73d0f80062f parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16947c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8ab713-fb8e-4fb3-968b-0a3ad6055783_isrForCrosstalkSources_1031139_41.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8ab713-fb8e-4fb3-968b-0a3ad6055783_isrForCrosstalkSources_1031139_41.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 98c595c92c517ab7f4cb8d7f1b2e6887 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/caff82c9-72f9-4c98-a45d-d20c9c67b455_isrForCrosstalkSources_1031141_45.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/caff82c9-72f9-4c98-a45d-d20c9c67b455_isrForCrosstalkSources_1031141_45.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 9d631dc1c87b096d03844da6c194dbda parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16949a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/df83a869-383a-409a-ad2c-3383a6d49040_isrForCrosstalkSources_1031141_19.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/df83a869-383a-409a-ad2c-3383a6d49040_isrForCrosstalkSources_1031141_19.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash f08c496469af79543bb7652a8d7b4900 parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d6fe5f50-5b02-4c9e-bba0-d56a358c5223_isrForCrosstalkSources_1031141_53.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d6fe5f50-5b02-4c9e-bba0-d56a358c5223_isrForCrosstalkSources_1031141_53.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 2a71577cb98a238bd974551aaa8c7349 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/32f7b5c2-76a4-4379-ad96-08fce246a775_isrForCrosstalkSources_1031141_38.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/32f7b5c2-76a4-4379-ad96-08fce246a775_isrForCrosstalkSources_1031141_38.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 993f046211bb1697413fc2b488c863d0 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bbda9e4d-881b-4b1f-b3c6-29c6ed16c225_isrForCrosstalkSources_1031140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bbda9e4d-881b-4b1f-b3c6-29c6ed16c225_isrForCrosstalkSources_1031140_24.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash b99c1602c70a55b8452c7ab825f4a54f parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/009a647e-405e-4d12-a63c-16fbfd6b18c0_isrForCrosstalkSources_1031141_60.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/009a647e-405e-4d12-a63c-16fbfd6b18c0_isrForCrosstalkSources_1031141_60.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 13097a8a529ceda7a5e017ce2e5cab68 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/284569b5-c39f-4735-8dae-6d58d735939f_isrForCrosstalkSources_1031141_34.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/284569b5-c39f-4735-8dae-6d58d735939f_isrForCrosstalkSources_1031141_34.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash bd5ae84eef423d7243392a12df66c95c parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/dd40c1e4-6ada-4e51-b4b8-d8ccfb9d7892_isrForCrosstalkSources_1031141_56.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/dd40c1e4-6ada-4e51-b4b8-d8ccfb9d7892_isrForCrosstalkSources_1031141_56.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 5c54b9680343284d5793b415a4a6916a parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/026ff445-2e8a-468c-91ad-b26b1fdd8af4_isrForCrosstalkSources_1031140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/026ff445-2e8a-468c-91ad-b26b1fdd8af4_isrForCrosstalkSources_1031140_10.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash fee5d2701cd4d56a1634ee679a91c75f parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/92edf746-a6f8-48b3-9136-23a002d62398_isrForCrosstalkSources_1031140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/92edf746-a6f8-48b3-9136-23a002d62398_isrForCrosstalkSources_1031140_29.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash 16b4bddd637b52aedf32cc81d4582a6b parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/1e38b595-244f-48c2-bdc0-2ca807099f3f_isrForCrosstalkSources_1031139_25.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/1e38b595-244f-48c2-bdc0-2ca807099f3f_isrForCrosstalkSources_1031139_25.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 72895f26435f65587fe5f8e9d087cdfa parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1694fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3a3f072f-6c30-4e5b-86a0-ad897f67f278_isrForCrosstalkSources_1031139_9.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3a3f072f-6c30-4e5b-86a0-ad897f67f278_isrForCrosstalkSources_1031139_9.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash 86092eb9888d72a8b9049ae60f84a1f3 parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/25bfa358-c0a6-4ba7-9123-1c0fcce4521f_isrForCrosstalkSources_1031139_19.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/25bfa358-c0a6-4ba7-9123-1c0fcce4521f_isrForCrosstalkSources_1031139_19.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash a398627558af4931047700db1b5eb19f parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16951c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/89466e0d-819a-4ce0-9cff-bd7ecaae0c44_isrForCrosstalkSources_1031140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/89466e0d-819a-4ce0-9cff-bd7ecaae0c44_isrForCrosstalkSources_1031140_2.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash ec86908268ab727388e8d1e66bf46f76 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/657eb189-78c6-4a33-aa7d-272a685003bd_isrForCrosstalkSources_1031139_61.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/657eb189-78c6-4a33-aa7d-272a685003bd_isrForCrosstalkSources_1031139_61.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 4d04bbc4b4d003eba830101583aabb52 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b27c7c04-10e4-4646-98ca-a3538662ff13_isrForCrosstalkSources_1031139_32.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b27c7c04-10e4-4646-98ca-a3538662ff13_isrForCrosstalkSources_1031139_32.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash a43197bff1b57a7d99ee994acf6f4597 parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6b55a1fd-50c9-444e-b404-9a934bdc10e5_isrForCrosstalkSources_1031141_30.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6b55a1fd-50c9-444e-b404-9a934bdc10e5_isrForCrosstalkSources_1031141_30.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 33761f6d614d004ac81b3cdbc037e9e5 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16953a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0230f5c6-af37-4a03-83ab-52bf7eea8f59_isrForCrosstalkSources_1031139_23.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0230f5c6-af37-4a03-83ab-52bf7eea8f59_isrForCrosstalkSources_1031139_23.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash ed20652857db26455f597240574bdf91 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16954e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a3141917-be51-4a8b-8045-0c3cca74fcf3_isrForCrosstalkSources_1031140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a3141917-be51-4a8b-8045-0c3cca74fcf3_isrForCrosstalkSources_1031140_54.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 7e2c787eac0b78ba321a221ad76c35eb parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/08a54dba-8e81-45fb-916e-7a0ca2df499a_isrForCrosstalkSources_1031141_29.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/08a54dba-8e81-45fb-916e-7a0ca2df499a_isrForCrosstalkSources_1031141_29.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 49094232bc1ebbe0f998483956ea61cd parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2ce7d325-0fc5-4697-b84b-3cb06e6534b0_isrForCrosstalkSources_1031140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2ce7d325-0fc5-4697-b84b-3cb06e6534b0_isrForCrosstalkSources_1031140_43.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 3d0438278c3d06569efa142810eed45c parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/bff59b81-b99d-4fd7-9579-151f8be10e93_isrForCrosstalkSources_1031141_23.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/bff59b81-b99d-4fd7-9579-151f8be10e93_isrForCrosstalkSources_1031141_23.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 8d04388390c56e2ef1cc613cce7d0810 parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16956c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8cd73728-c1af-4855-b954-66c69cee6eaf_isrForCrosstalkSources_1031140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8cd73728-c1af-4855-b954-66c69cee6eaf_isrForCrosstalkSources_1031140_35.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 55905caa6198c51c257838a797164b0f parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5124a823-55ff-4292-a913-c05bf554b4de_isrForCrosstalkSources_1031141_43.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5124a823-55ff-4292-a913-c05bf554b4de_isrForCrosstalkSources_1031141_43.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 44651564911353e643a2b94e0f30cfca parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/390d8cbc-21b9-4cfd-826a-617982fc0593_isrForCrosstalkSources_1031139_60.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/390d8cbc-21b9-4cfd-826a-617982fc0593_isrForCrosstalkSources_1031139_60.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash 39a1b937d78e73660edbcb01863e20e6 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16958a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a3025387-8fd2-444b-9979-cd02df404d0f_isrForCrosstalkSources_1031139_18.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a3025387-8fd2-444b-9979-cd02df404d0f_isrForCrosstalkSources_1031139_18.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 601c55ed7945c4f31a3f3d4ee4bb1945 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bb730d9b-399f-48e5-96dc-c54ee6fe1a3a_isrForCrosstalkSources_1031140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bb730d9b-399f-48e5-96dc-c54ee6fe1a3a_isrForCrosstalkSources_1031140_28.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash e0639441bb1616bbcb92a4ceff5dd9c9 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16959e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/481a3d0f-2a3d-459a-8f06-7d19e5222eb0_isrForCrosstalkSources_1031140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/481a3d0f-2a3d-459a-8f06-7d19e5222eb0_isrForCrosstalkSources_1031140_52.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 7617cc59f74d0de84498e2d15dfaa04f parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/5fbce5d9-0de2-4184-a057-4abe39ae2f5d_isrForCrosstalkSources_1031139_47.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/5fbce5d9-0de2-4184-a057-4abe39ae2f5d_isrForCrosstalkSources_1031139_47.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 2d4ebcdba2a50b18b825e6ae696ebbe5 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/31f99e87-724f-4bf6-889c-128845b23fdd_isrForCrosstalkSources_1031139_13.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/31f99e87-724f-4bf6-889c-128845b23fdd_isrForCrosstalkSources_1031139_13.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash b3f4ccbc4623015049fa65ddafe0bc99 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-17 01:26:18 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3e2fd1f3-f825-4423-bb79-9b1d28b23a72_isrForCrosstalkSources_1031139_58.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3e2fd1f3-f825-4423-bb79-9b1d28b23a72_isrForCrosstalkSources_1031139_58.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 34c8f8334113724fc8548a411f5cf5e3 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a9f8c944-469c-40bf-8c26-dba4ffe66b7a_isrForCrosstalkSources_1031140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a9f8c944-469c-40bf-8c26-dba4ffe66b7a_isrForCrosstalkSources_1031140_8.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 4dad1d726405811d913b1da03dc829dc parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/819087f3-d146-4ec8-8971-69ca09b5093e_isrForCrosstalkSources_1031141_41.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/819087f3-d146-4ec8-8971-69ca09b5093e_isrForCrosstalkSources_1031141_41.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash e58040a2ec90811346725539930ee410 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16960c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6cd59e70-2789-48cd-9c3e-a66ab931131b_isrForCrosstalkSources_1031141_50.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6cd59e70-2789-48cd-9c3e-a66ab931131b_isrForCrosstalkSources_1031141_50.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 7e6b44ce3c619d01afd052028c12ffcc parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/a448fe8b-065f-45eb-9dd2-44256ff9a61e_isrForCrosstalkSources_1031141_8.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/a448fe8b-065f-45eb-9dd2-44256ff9a61e_isrForCrosstalkSources_1031141_8.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash a96b74f913aea1b253eceecec35afb15 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/453bc840-eb65-4245-949a-7284c0cc6d6a_isrForCrosstalkSources_1031141_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/453bc840-eb65-4245-949a-7284c0cc6d6a_isrForCrosstalkSources_1031141_61.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash ad1731b17d3636849c56eb88f07fdf2e parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/123f03cd-b781-4362-b312-619008c6a691_isrForCrosstalkSources_1031141_18.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/123f03cd-b781-4362-b312-619008c6a691_isrForCrosstalkSources_1031141_18.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 741e7d0acf6882dfa600f4429c2e9b9c parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e14259c3-4e45-4674-98b7-5ab1d06341d0_isrForCrosstalkSources_1031141_14.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e14259c3-4e45-4674-98b7-5ab1d06341d0_isrForCrosstalkSources_1031141_14.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash f8edda8e1eac56942bf6414347b825d6 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/657f9cd4-8e0e-42e2-b7a6-7227bf940a80_isrForCrosstalkSources_1031140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/657f9cd4-8e0e-42e2-b7a6-7227bf940a80_isrForCrosstalkSources_1031140_1.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 8a8fce1e0a2ccfae10fecb10d5892a28 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/d3e2cd45-fdd8-479c-acd1-7335c187a698_isrForCrosstalkSources_1031140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/d3e2cd45-fdd8-479c-acd1-7335c187a698_isrForCrosstalkSources_1031140_7.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 8e02bd194640a06e77f28bcc5552d5f0 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1695f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5f162c38-37d0-49b2-8b68-1b90d8acaa07_isrForCrosstalkSources_1031140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5f162c38-37d0-49b2-8b68-1b90d8acaa07_isrForCrosstalkSources_1031140_42.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash a261db90a70aefa827e52d074da6ebc7 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16962a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 161 parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/77dc4262-1bdc-4dda-b7f4-7267e3f140c0_isrForCrosstalkSources_1031140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/77dc4262-1bdc-4dda-b7f4-7267e3f140c0_isrForCrosstalkSources_1031140_18.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash cffde0d27712b34bf46f88763375e72f parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/90539147-a963-409b-ab67-719377ca86ba_isrForCrosstalkSources_1031139_5.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/90539147-a963-409b-ab67-719377ca86ba_isrForCrosstalkSources_1031139_5.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash 119202ed25eac9e23bc69ed9ff8c1b64 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fbb7a654-6be2-4a3a-a137-7686c31d39a8_isrForCrosstalkSources_1031140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fbb7a654-6be2-4a3a-a137-7686c31d39a8_isrForCrosstalkSources_1031140_59.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 890d51aec8f7d43a419b35431e43974c parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16963e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Executor multi has 161 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 40 more blocks parsl.executors.status_handling INFO: Scaling out by 40 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-17 01:26:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/ae165f66-9705-4362-9e3a-7cab76c79c7d_isrForCrosstalkSources_1031140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/ae165f66-9705-4362-9e3a-7cab76c79c7d_isrForCrosstalkSources_1031140_17.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 509f6315278effa9d57767a062975742 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8fc56329-3b40-4518-a90d-3cce51d31ed7_isrForCrosstalkSources_1031141_27.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8fc56329-3b40-4518-a90d-3cce51d31ed7_isrForCrosstalkSources_1031141_27.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 8a3a297eb7cd23d41e364b242a051aa3 parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16965c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61c62b4a-5eac-4404-8ae5-c57d7cb3d988_isrForCrosstalkSources_1031140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61c62b4a-5eac-4404-8ae5-c57d7cb3d988_isrForCrosstalkSources_1031140_38.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 140c65b4735487e130d5a5a6959100d1 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16968e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0ea47cff-c54f-4111-b578-5e45eff8d333_isrForCrosstalkSources_1031141_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0ea47cff-c54f-4111-b578-5e45eff8d333_isrForCrosstalkSources_1031141_31.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash ffd02bb22fd1f03f604e5f18886d1733 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/4cc63008-cbd6-4e1b-b76e-f019787f44d8_isrForCrosstalkSources_1031140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/4cc63008-cbd6-4e1b-b76e-f019787f44d8_isrForCrosstalkSources_1031140_37.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 70f6f946db7c3d4c6e93c4bdce173424 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7817581e-4f75-4b15-ae9b-1dbddcc93f05_isrForCrosstalkSources_1031141_21.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7817581e-4f75-4b15-ae9b-1dbddcc93f05_isrForCrosstalkSources_1031141_21.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash 5453bd57481c3b9092c716a08b6b1545 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49a520ae-744d-4179-854b-3a5f363a11f6_isrForCrosstalkSources_1031139_59.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49a520ae-744d-4179-854b-3a5f363a11f6_isrForCrosstalkSources_1031139_59.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash de2e79a6f6c30e546affea3f02f92e15 parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/330c310d-979c-4d0e-9ebd-1c7e13325555_isrForCrosstalkSources_1031140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/330c310d-979c-4d0e-9ebd-1c7e13325555_isrForCrosstalkSources_1031140_21.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 1d148a20ea571990d8b3af09954a7903 parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c3f41142-061a-41c8-b303-70b29c5c8561_isrForCrosstalkSources_1031141_35.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c3f41142-061a-41c8-b303-70b29c5c8561_isrForCrosstalkSources_1031141_35.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 9e2c2207773dbfb7cf103e269d051479 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eec9986-4587-4d3f-bd6a-228395d7d3b7_isrForCrosstalkSources_1031140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eec9986-4587-4d3f-bd6a-228395d7d3b7_isrForCrosstalkSources_1031140_55.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash c1fc1405fd29ac2fbe76fdb2c15a1e27 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1697060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b7f01cdf-29d1-4d1b-b28f-05e9f3c246cb_isrForCrosstalkSources_1031139_17.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b7f01cdf-29d1-4d1b-b28f-05e9f3c246cb_isrForCrosstalkSources_1031139_17.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash bae4983cbbeac51ec162a7aef5ceccb6 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4dc8b663-05ae-474a-9926-1b1ad3512173_isrForCrosstalkSources_1031141_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4dc8b663-05ae-474a-9926-1b1ad3512173_isrForCrosstalkSources_1031141_58.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 0f9692d4f358ad4dfe0f2f8e0832ae9e parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1696fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/31c2f8fb-38e9-4e3e-9fbf-47e89a472de2_isrForCrosstalkSources_1031140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/31c2f8fb-38e9-4e3e-9fbf-47e89a472de2_isrForCrosstalkSources_1031140_5.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 4e14e7bcd821f91a56da22b56f6e70c5 parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1697100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1a6683aa-9dd5-4672-9412-cb301cf9404f_isrForCrosstalkSources_1031140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1a6683aa-9dd5-4672-9412-cb301cf9404f_isrForCrosstalkSources_1031140_31.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash f9cfeff95580c784a606888b12a18394 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1697240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e3fb814a-c1f0-4804-9e18-e854ebf8cb92_isrForCrosstalkSources_1031140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e3fb814a-c1f0-4804-9e18-e854ebf8cb92_isrForCrosstalkSources_1031140_22.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash c52735dc4efa8a5ddb16c21fbb47bc23 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1697380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/675d1bc9-9573-4aa5-aa3e-1b2e8e9174a8_isrForCrosstalkSources_1031141_59.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/675d1bc9-9573-4aa5-aa3e-1b2e8e9174a8_isrForCrosstalkSources_1031141_59.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 01a5b3697f9a1cdd00fc4b0501262dac parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16971a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c7a7b2ec-9172-49e6-a690-e37ad4d13889_isrForCrosstalkSources_1031141_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c7a7b2ec-9172-49e6-a690-e37ad4d13889_isrForCrosstalkSources_1031141_9.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash 8d12a182b16ac573b1caf36f7eed16b0 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16972e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d546f6a7-a708-4c23-8c25-b2f279a9aea1_isrForCrosstalkSources_1031141_54.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d546f6a7-a708-4c23-8c25-b2f279a9aea1_isrForCrosstalkSources_1031141_54.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 2fcc8fe3fd99babfa04f66b47c690b71 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a16974c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/61f9e92c-f27f-4660-bb75-ca5bbeac4ede_isrForCrosstalkSources_1031139_62.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/61f9e92c-f27f-4660-bb75-ca5bbeac4ede_isrForCrosstalkSources_1031139_62.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 819f6110f59f35200b1fbe1371b3a886 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1501a1697600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f96c9810-b81e-42ba-83a2-44751020a7ed_isrForCrosstalkSources_1031140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f96c9810-b81e-42ba-83a2-44751020a7ed_isrForCrosstalkSources_1031140_47.stderr 2024-03-17 01:26:19 proc_lsst.multi:170 [INFO] [multi] job_id 17142638 proc_lsst.multi INFO: [multi] job_id 17142638 2024-03-17 01:26:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142638 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142638 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17142638 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-17 01:26:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:19 proc_lsst.multi:170 [INFO] [multi] job_id 17142639 proc_lsst.multi INFO: [multi] job_id 17142639 2024-03-17 01:26:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142639 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142639 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17142639 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-17 01:26:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 17142640 proc_lsst.multi INFO: [multi] job_id 17142640 2024-03-17 01:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142640 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142640 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17142640 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-17 01:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 17142641 proc_lsst.multi INFO: [multi] job_id 17142641 2024-03-17 01:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142641 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142641 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17142641 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-17 01:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 17142642 proc_lsst.multi INFO: [multi] job_id 17142642 2024-03-17 01:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142642 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142642 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17142642 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-17 01:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 17142643 proc_lsst.multi INFO: [multi] job_id 17142643 2024-03-17 01:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 01:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142643 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142643 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17142643 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-17 01:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 01:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:21 proc_lsst.multi:170 [INFO] [multi] job_id 17142644 proc_lsst.multi INFO: [multi] job_id 17142644 2024-03-17 01:26:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 01:26:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142644 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142644 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17142644 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-17 01:26:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-17 01:26:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 01:26:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:21 proc_lsst.multi:170 [INFO] [multi] job_id 17142645 proc_lsst.multi INFO: [multi] job_id 17142645 2024-03-17 01:26:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 01:26:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142645 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142645 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17142645 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-17 01:26:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-17 01:26:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 01:26:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:22 proc_lsst.multi:170 [INFO] [multi] job_id 17142646 proc_lsst.multi INFO: [multi] job_id 17142646 2024-03-17 01:26:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 01:26:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142646 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142646 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17142646 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-17 01:26:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:22 proc_lsst.multi:170 [INFO] [multi] job_id 17142647 proc_lsst.multi INFO: [multi] job_id 17142647 2024-03-17 01:26:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 01:26:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142647 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142647 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17142647 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-17 01:26:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:22 proc_lsst.multi:170 [INFO] [multi] job_id 17142648 proc_lsst.multi INFO: [multi] job_id 17142648 2024-03-17 01:26:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 01:26:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142648 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142648 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17142648 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-17 01:26:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-17 01:26:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:23 proc_lsst.multi:170 [INFO] [multi] job_id 17142650 proc_lsst.multi INFO: [multi] job_id 17142650 2024-03-17 01:26:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 01:26:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142650 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142650 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17142650 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-17 01:26:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-17 01:26:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-17 01:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:23 proc_lsst.multi:170 [INFO] [multi] job_id 17142651 proc_lsst.multi INFO: [multi] job_id 17142651 2024-03-17 01:26:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 01:26:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:23 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142651 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142651 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17142651 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-17 01:26:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-17 01:26:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-17 01:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 17142652 proc_lsst.multi INFO: [multi] job_id 17142652 2024-03-17 01:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 01:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142652 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142652 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17142652 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-17 01:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 17142653 proc_lsst.multi INFO: [multi] job_id 17142653 2024-03-17 01:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 01:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142653 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142653 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17142653 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-17 01:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 17142654 proc_lsst.multi INFO: [multi] job_id 17142654 2024-03-17 01:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 01:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142654 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142654 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17142654 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-17 01:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 17142655 proc_lsst.multi INFO: [multi] job_id 17142655 2024-03-17 01:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 01:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142655 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142655 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17142655 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-17 01:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 17142656 proc_lsst.multi INFO: [multi] job_id 17142656 2024-03-17 01:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 01:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142656 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142656 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17142656 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-17 01:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-17 01:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:25 proc_lsst.multi:170 [INFO] [multi] job_id 17142657 proc_lsst.multi INFO: [multi] job_id 17142657 2024-03-17 01:26:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 01:26:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142657 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142657 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17142657 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-17 01:26:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:25 proc_lsst.multi:170 [INFO] [multi] job_id 17142658 proc_lsst.multi INFO: [multi] job_id 17142658 2024-03-17 01:26:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 01:26:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142658 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142658 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17142658 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-17 01:26:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:25 proc_lsst.multi:170 [INFO] [multi] job_id 17142659 proc_lsst.multi INFO: [multi] job_id 17142659 2024-03-17 01:26:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 01:26:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142659 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142659 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17142659 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-17 01:26:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-17 01:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:26 proc_lsst.multi:170 [INFO] [multi] job_id 17142660 proc_lsst.multi INFO: [multi] job_id 17142660 2024-03-17 01:26:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 01:26:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142660 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142660 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17142660 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-17 01:26:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:26 proc_lsst.multi:170 [INFO] [multi] job_id 17142661 proc_lsst.multi INFO: [multi] job_id 17142661 2024-03-17 01:26:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 01:26:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142661 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142661 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17142661 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-17 01:26:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:26 proc_lsst.multi:170 [INFO] [multi] job_id 17142662 proc_lsst.multi INFO: [multi] job_id 17142662 2024-03-17 01:26:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 01:26:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142662 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142662 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17142662 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-17 01:26:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-17 01:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:27 proc_lsst.multi:170 [INFO] [multi] job_id 17142663 proc_lsst.multi INFO: [multi] job_id 17142663 2024-03-17 01:26:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 01:26:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142663 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142663 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17142663 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-17 01:26:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:27 proc_lsst.multi:170 [INFO] [multi] job_id 17142664 proc_lsst.multi INFO: [multi] job_id 17142664 2024-03-17 01:26:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 01:26:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142664 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142664 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17142664 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-17 01:26:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:27 proc_lsst.multi:170 [INFO] [multi] job_id 17142665 proc_lsst.multi INFO: [multi] job_id 17142665 2024-03-17 01:26:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 01:26:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142665 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142665 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17142665 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-17 01:26:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-17 01:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:28 proc_lsst.multi:170 [INFO] [multi] job_id 17142666 proc_lsst.multi INFO: [multi] job_id 17142666 2024-03-17 01:26:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 01:26:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142666 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142666 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17142666 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-17 01:26:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-17 01:26:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-17 01:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:28 proc_lsst.multi:170 [INFO] [multi] job_id 17142667 proc_lsst.multi INFO: [multi] job_id 17142667 2024-03-17 01:26:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 01:26:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:28 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142667 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142667 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17142667 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-17 01:26:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-17 01:26:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-17 01:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62454ff9da77a39e6e034c383cc726cc with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4131bd7f-78b6-43f9-bd96-eef40eb21027_isrForCrosstalkSources_1031141_7.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4131bd7f-78b6-43f9-bd96-eef40eb21027_isrForCrosstalkSources_1031141_7.stderr 2024-03-17 01:26:29 proc_lsst.multi:170 [INFO] [multi] job_id 17142668 proc_lsst.multi INFO: [multi] job_id 17142668 2024-03-17 01:26:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:29 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17142668 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17142668 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17142668 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-17 01:26:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-17 01:26:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 01:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de594ad26492f5b22081fe3550680b0 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/c2b542e9-8aa1-4edb-88a2-138f67b82cfa_isrForCrosstalkSources_1031139_52.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/c2b542e9-8aa1-4edb-88a2-138f67b82cfa_isrForCrosstalkSources_1031139_52.stderr 2024-03-17 01:26:30 proc_lsst.multi:170 [INFO] [multi] job_id 17142669 proc_lsst.multi INFO: [multi] job_id 17142669 2024-03-17 01:26:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:30 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17142669 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17142669 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17142669 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-17 01:26:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-17 01:26:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:31 proc_lsst.multi:170 [INFO] [multi] job_id 17142670 proc_lsst.multi INFO: [multi] job_id 17142670 2024-03-17 01:26:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17142670 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17142670 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17142670 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-17 01:26:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:31 proc_lsst.multi:170 [INFO] [multi] job_id 17142671 proc_lsst.multi INFO: [multi] job_id 17142671 2024-03-17 01:26:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17142671 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17142671 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17142671 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-17 01:26:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:31 proc_lsst.multi:170 [INFO] [multi] job_id 17142672 proc_lsst.multi INFO: [multi] job_id 17142672 2024-03-17 01:26:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17142672 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17142672 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17142672 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-17 01:26:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:32 proc_lsst.multi:170 [INFO] [multi] job_id 17142673 proc_lsst.multi INFO: [multi] job_id 17142673 2024-03-17 01:26:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:32 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17142673 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17142673 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17142673 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-17 01:26:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-17 01:26:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-17 01:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:33 proc_lsst.multi:170 [INFO] [multi] job_id 17142674 proc_lsst.multi INFO: [multi] job_id 17142674 2024-03-17 01:26:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142674 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142674 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17142674 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-17 01:26:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:33 proc_lsst.multi:170 [INFO] [multi] job_id 17142675 proc_lsst.multi INFO: [multi] job_id 17142675 2024-03-17 01:26:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:33 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142675 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142675 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17142675 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-17 01:26:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-17 01:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:34 proc_lsst.multi:170 [INFO] [multi] job_id 17142676 proc_lsst.multi INFO: [multi] job_id 17142676 2024-03-17 01:26:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142676 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142676 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17142676 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-17 01:26:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:34 proc_lsst.multi:170 [INFO] [multi] job_id 17142677 proc_lsst.multi INFO: [multi] job_id 17142677 2024-03-17 01:26:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142677 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142677 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17142677 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-17 01:26:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:34 proc_lsst.multi:170 [INFO] [multi] job_id 17142678 proc_lsst.multi INFO: [multi] job_id 17142678 2024-03-17 01:26:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142678 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142678 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17142678 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 41, active_tasks = 184 parsl.dataflow.strategy DEBUG: Executor multi has 184 active tasks, 1/40 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 41 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 5 more blocks parsl.executors.status_handling INFO: Scaling out by 5 blocks parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-17 01:26:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:35 proc_lsst.multi:170 [INFO] [multi] job_id 17142683 proc_lsst.multi INFO: [multi] job_id 17142683 2024-03-17 01:26:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 01:26:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142683 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142683 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17142683 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-17 01:26:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:35 proc_lsst.multi:170 [INFO] [multi] job_id 17142687 proc_lsst.multi INFO: [multi] job_id 17142687 2024-03-17 01:26:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 01:26:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142687 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142687 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17142687 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-17 01:26:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:35 proc_lsst.multi:170 [INFO] [multi] job_id 17142688 proc_lsst.multi INFO: [multi] job_id 17142688 2024-03-17 01:26:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 01:26:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142688 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142688 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17142688 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-17 01:26:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-17 01:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:36 proc_lsst.multi:170 [INFO] [multi] job_id 17142689 proc_lsst.multi INFO: [multi] job_id 17142689 2024-03-17 01:26:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 01:26:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142689 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142689 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17142689 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-17 01:26:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-17 01:26:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-17 01:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-17 01:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-17 01:26:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-17 01:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-17 01:26:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-17 01:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-17 01:26:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-17 01:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-17 01:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 198.48.92.26,10.64.65.9,10.64.129.9,127.0.0.1,169.254.95.120,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54703 --result_port=54752 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-17 01:26:36 proc_lsst.multi:170 [INFO] [multi] job_id 17142690 proc_lsst.multi INFO: [multi] job_id 17142690 2024-03-17 01:26:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-17 01:26:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-17 01:26:36 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17142690 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17142690 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17142690 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03d09b674def21d7d1229937c6447b32 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/19e9311d-923e-403e-8f1e-c2b9b371120c_isrForCrosstalkSources_1031141_51.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/19e9311d-923e-403e-8f1e-c2b9b371120c_isrForCrosstalkSources_1031141_51.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 183 parsl.dataflow.strategy DEBUG: Executor multi has 183 active tasks, 1/45 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 721961918518914f88079382851800ef with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/88683b2b-a5bc-416c-9aed-55f8980ef482_isrForCrosstalkSources_1031140_30.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/88683b2b-a5bc-416c-9aed-55f8980ef482_isrForCrosstalkSources_1031140_30.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 182 parsl.dataflow.strategy DEBUG: Executor multi has 182 active tasks, 1/45 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f24b23f9260a03f3d8f82555a4702b with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9623c7c4-6505-4f2b-a148-398263ebf086_isrForCrosstalkSources_1031140_27.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9623c7c4-6505-4f2b-a148-398263ebf086_isrForCrosstalkSources_1031140_27.stderr 2024-03-17 01:26:49 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local 2024-03-17 01:26:49 proc_lsst.multi:146 [INFO] found job 17142638 in provider astro proc_lsst.multi INFO: found job 17142638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:50 proc_lsst.multi:146 [INFO] found job 17142639 in provider astro proc_lsst.multi INFO: found job 17142639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:50 proc_lsst.multi:146 [INFO] found job 17142640 in provider astro proc_lsst.multi INFO: found job 17142640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:51 proc_lsst.multi:146 [INFO] found job 17142641 in provider astro proc_lsst.multi INFO: found job 17142641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c122fe9dcd5a54c49ef66801d430cc6e with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/be869998-97bf-4931-b86b-70d912e6de51_isrForCrosstalkSources_1031139_35.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/be869998-97bf-4931-b86b-70d912e6de51_isrForCrosstalkSources_1031139_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:51 proc_lsst.multi:146 [INFO] found job 17142642 in provider astro proc_lsst.multi INFO: found job 17142642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:51 proc_lsst.multi:146 [INFO] found job 17142643 in provider astro proc_lsst.multi INFO: found job 17142643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:52 proc_lsst.multi:146 [INFO] found job 17142644 in provider astro proc_lsst.multi INFO: found job 17142644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:52 proc_lsst.multi:146 [INFO] found job 17142645 in provider astro proc_lsst.multi INFO: found job 17142645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:53 proc_lsst.multi:146 [INFO] found job 17142646 in provider astro proc_lsst.multi INFO: found job 17142646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:54 proc_lsst.multi:146 [INFO] found job 17142647 in provider astro proc_lsst.multi INFO: found job 17142647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:54 proc_lsst.multi:146 [INFO] found job 17142648 in provider astro proc_lsst.multi INFO: found job 17142648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69dc938499b9212d340a3c816b8f266f with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/44766ab3-4011-4052-b7a2-a2db97b95568_isrForCrosstalkSources_1031140_3.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/44766ab3-4011-4052-b7a2-a2db97b95568_isrForCrosstalkSources_1031140_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:55 proc_lsst.multi:146 [INFO] found job 17142650 in provider astro proc_lsst.multi INFO: found job 17142650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:55 proc_lsst.multi:146 [INFO] found job 17142651 in provider astro proc_lsst.multi INFO: found job 17142651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:56 proc_lsst.multi:146 [INFO] found job 17142652 in provider astro proc_lsst.multi INFO: found job 17142652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:56 proc_lsst.multi:146 [INFO] found job 17142653 in provider astro proc_lsst.multi INFO: found job 17142653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:57 proc_lsst.multi:146 [INFO] found job 17142654 in provider astro proc_lsst.multi INFO: found job 17142654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:57 proc_lsst.multi:146 [INFO] found job 17142655 in provider astro proc_lsst.multi INFO: found job 17142655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:58 proc_lsst.multi:146 [INFO] found job 17142656 in provider astro proc_lsst.multi INFO: found job 17142656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:58 proc_lsst.multi:146 [INFO] found job 17142657 in provider astro proc_lsst.multi INFO: found job 17142657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:59 proc_lsst.multi:146 [INFO] found job 17142658 in provider astro proc_lsst.multi INFO: found job 17142658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:26:59 proc_lsst.multi:146 [INFO] found job 17142659 in provider astro proc_lsst.multi INFO: found job 17142659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8a6a185618f4f68eef9c1131741b183 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e07c859a-d2fb-4353-a5a9-0472ccf365b4_isrForCrosstalkSources_1031140_36.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e07c859a-d2fb-4353-a5a9-0472ccf365b4_isrForCrosstalkSources_1031140_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:00 proc_lsst.multi:146 [INFO] found job 17142660 in provider astro proc_lsst.multi INFO: found job 17142660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:01 proc_lsst.multi:146 [INFO] found job 17142661 in provider astro proc_lsst.multi INFO: found job 17142661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:02 proc_lsst.multi:146 [INFO] found job 17142662 in provider astro proc_lsst.multi INFO: found job 17142662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:02 proc_lsst.multi:146 [INFO] found job 17142663 in provider astro proc_lsst.multi INFO: found job 17142663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:03 proc_lsst.multi:146 [INFO] found job 17142664 in provider astro proc_lsst.multi INFO: found job 17142664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:04 proc_lsst.multi:146 [INFO] found job 17142665 in provider astro proc_lsst.multi INFO: found job 17142665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:04 proc_lsst.multi:146 [INFO] found job 17142666 in provider astro proc_lsst.multi INFO: found job 17142666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:05 proc_lsst.multi:146 [INFO] found job 17142667 in provider astro proc_lsst.multi INFO: found job 17142667 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:05 proc_lsst.multi:146 [INFO] found job 17142668 in provider astro proc_lsst.multi INFO: found job 17142668 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b6bd9cdea8d796b14245d1e601e99e1 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3ba8d41c-c3c3-4ef6-972b-04e04d2f20ca_isrForCrosstalkSources_1031139_29.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3ba8d41c-c3c3-4ef6-972b-04e04d2f20ca_isrForCrosstalkSources_1031139_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:05 proc_lsst.multi:146 [INFO] found job 17142669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:06 proc_lsst.multi:146 [INFO] found job 17142670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:07 proc_lsst.multi:146 [INFO] found job 17142671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:07 proc_lsst.multi:146 [INFO] found job 17142672 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142672 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:08 proc_lsst.multi:146 [INFO] found job 17142673 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142673 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:08 proc_lsst.multi:146 [INFO] found job 17142674 in provider escience_a40 proc_lsst.multi INFO: found job 17142674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:09 proc_lsst.multi:146 [INFO] found job 17142675 in provider escience_a40 proc_lsst.multi INFO: found job 17142675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:09 proc_lsst.multi:146 [INFO] found job 17142676 in provider escience_a40 proc_lsst.multi INFO: found job 17142676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d7d3e9dae0bf494ab93f1cd871003d8 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6ed4700e-18e7-4523-9415-bb8de1a6c156_isrForCrosstalkSources_1031139_12.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6ed4700e-18e7-4523-9415-bb8de1a6c156_isrForCrosstalkSources_1031139_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:10 proc_lsst.multi:146 [INFO] found job 17142677 in provider escience_a40 proc_lsst.multi INFO: found job 17142677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:10 proc_lsst.multi:146 [INFO] found job 17142678 in provider escience_a40 proc_lsst.multi INFO: found job 17142678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:10 proc_lsst.multi:146 [INFO] found job 17142683 in provider escience_a40 proc_lsst.multi INFO: found job 17142683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:11 proc_lsst.multi:146 [INFO] found job 17142687 in provider escience_a40 proc_lsst.multi INFO: found job 17142687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:11 proc_lsst.multi:146 [INFO] found job 17142688 in provider escience_a40 proc_lsst.multi INFO: found job 17142688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:12 proc_lsst.multi:146 [INFO] found job 17142689 in provider escience_a40 proc_lsst.multi INFO: found job 17142689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:12 proc_lsst.multi:146 [INFO] found job 17142690 in provider escience_a40 proc_lsst.multi INFO: found job 17142690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 176 parsl.dataflow.strategy DEBUG: Executor multi has 176 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 176 parsl.dataflow.strategy DEBUG: Executor multi has 176 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65264c6630b29815b359ed99831fd62b with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fc9e25f1-a459-4eb4-9a05-110412ea086f_isrForCrosstalkSources_1031140_19.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fc9e25f1-a459-4eb4-9a05-110412ea086f_isrForCrosstalkSources_1031140_19.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1c6d57d5b9f5d97cda0a4865bd81325 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/0ac8a91a-8d99-4b3d-ba82-70bc5b9d72a0_isrForCrosstalkSources_1031140_46.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/0ac8a91a-8d99-4b3d-ba82-70bc5b9d72a0_isrForCrosstalkSources_1031140_46.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a86e53d6fcc6d382dbc23c0f6aa5a30 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b65f50d3-3a6a-41bc-a4b0-da584f0a883f_isrForCrosstalkSources_1031140_16.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b65f50d3-3a6a-41bc-a4b0-da584f0a883f_isrForCrosstalkSources_1031140_16.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7ce08c2f49eb78b72b21ada43ddb3bdc with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8540ccb9-0865-4ab2-a75c-31e1d35668e6_isrForCrosstalkSources_1031139_22.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8540ccb9-0865-4ab2-a75c-31e1d35668e6_isrForCrosstalkSources_1031139_22.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 172 parsl.dataflow.strategy DEBUG: Executor multi has 172 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d62f2b7dcb420d9c328046d16bbdf114 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/821ebf1a-e7e3-4863-8a93-0f4bf10010f4_isrForCrosstalkSources_1031141_3.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/821ebf1a-e7e3-4863-8a93-0f4bf10010f4_isrForCrosstalkSources_1031141_3.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2b78d16c2906cf6146fae64238492d with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a142e785-3642-4d04-9836-72a679d8b7f0_isrForCrosstalkSources_1031139_26.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a142e785-3642-4d04-9836-72a679d8b7f0_isrForCrosstalkSources_1031139_26.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b060f325ecb3cc1d90d457464c3d71c6 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1bc6bf0d-999e-413e-bf9e-bc9d4f977238_isrForCrosstalkSources_1031141_2.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1bc6bf0d-999e-413e-bf9e-bc9d4f977238_isrForCrosstalkSources_1031141_2.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db9b36f70374020972e566410a8ef422 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9ea4f61f-a3a6-47e4-a28f-2011ae5a5552_isrForCrosstalkSources_1031140_9.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/9ea4f61f-a3a6-47e4-a28f-2011ae5a5552_isrForCrosstalkSources_1031140_9.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05636602d29c700e085324a186d80fec with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7f2c9697-eff7-4fd5-8b4e-ff7c9a19296c_isrForCrosstalkSources_1031139_4.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7f2c9697-eff7-4fd5-8b4e-ff7c9a19296c_isrForCrosstalkSources_1031139_4.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bae3575465f5634979349e07478897f with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4ebcb221-b3e3-404a-85a6-91ec0b33a3cf_isrForCrosstalkSources_1031139_21.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4ebcb221-b3e3-404a-85a6-91ec0b33a3cf_isrForCrosstalkSources_1031139_21.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 893a667290ac67b9685865e080d39a7e with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8a03fa65-740a-41bc-a27d-825f2d34b497_isrForCrosstalkSources_1031139_45.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8a03fa65-740a-41bc-a27d-825f2d34b497_isrForCrosstalkSources_1031139_45.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de20627cb3262dcd156c8fe5b8212489 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18ee3a5b-3583-495a-b726-b3c11b0c7536_isrForCrosstalkSources_1031141_36.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18ee3a5b-3583-495a-b726-b3c11b0c7536_isrForCrosstalkSources_1031141_36.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e10b5687332d76b0bf84561c97d406ea with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7db773be-a774-4c28-a384-d26a23d130b7_isrForCrosstalkSources_1031139_6.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/7db773be-a774-4c28-a384-d26a23d130b7_isrForCrosstalkSources_1031139_6.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b55a6dc2134c60f39df99ceb6d40b68c with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/63f808a9-fcf8-4f05-a863-43c52b14e9d5_isrForCrosstalkSources_1031141_48.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/63f808a9-fcf8-4f05-a863-43c52b14e9d5_isrForCrosstalkSources_1031141_48.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81a9c06d4717c5258a5e63239144565c with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/535fb1c2-815b-41d1-a0d3-959d2b5d3b75_isrForCrosstalkSources_1031140_13.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/535fb1c2-815b-41d1-a0d3-959d2b5d3b75_isrForCrosstalkSources_1031140_13.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1dcc72c18add70d142a67bf0f000695 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/91212434-6211-4358-ad0c-5fc6ecccab30_isrForCrosstalkSources_1031140_62.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/91212434-6211-4358-ad0c-5fc6ecccab30_isrForCrosstalkSources_1031140_62.stderr parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0cdcb17161b95d457afe6d150d453494 with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ed15aa3f-4871-4e85-880e-1eb94c03c1ad_isrForCrosstalkSources_1031141_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ed15aa3f-4871-4e85-880e-1eb94c03c1ad_isrForCrosstalkSources_1031141_5.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55d6d501ec93a3d467a34103dc8e34a0 with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/55980193-cf50-4a29-9093-95f788e7ccc5_isrForCrosstalkSources_1031139_33.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/55980193-cf50-4a29-9093-95f788e7ccc5_isrForCrosstalkSources_1031139_33.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19596a64d530be040f627b40f7b78bdb with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18321e31-cc38-43b4-88e3-d1e8fbb24269_isrForCrosstalkSources_1031141_37.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/18321e31-cc38-43b4-88e3-d1e8fbb24269_isrForCrosstalkSources_1031141_37.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 595fdef3ae55be533a715cae272ce5e9 with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2e291ab2-98db-4b85-94ab-3fd04a5613f7_isrForCrosstalkSources_1031140_51.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2e291ab2-98db-4b85-94ab-3fd04a5613f7_isrForCrosstalkSources_1031140_51.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6bbd1a663daa91c58542186c5b3b3a7 with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/ccaabdb5-4488-4f61-a960-fb03815e62ea_isrForCrosstalkSources_1031139_48.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/ccaabdb5-4488-4f61-a960-fb03815e62ea_isrForCrosstalkSources_1031139_48.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12293e80796796ec9a62e45dfb5ab7c3 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8a7e7a-c0c5-44df-bbf6-6ce86a71f506_isrForCrosstalkSources_1031139_24.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8a7e7a-c0c5-44df-bbf6-6ce86a71f506_isrForCrosstalkSources_1031139_24.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry caa0fa66cfe84e13f00b2ca0d2d6c9ea with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8abc552d-f01b-4cb4-831f-1eddba502851_isrForCrosstalkSources_1031141_12.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8abc552d-f01b-4cb4-831f-1eddba502851_isrForCrosstalkSources_1031141_12.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 781f6b6e0964d3f77b7811ab0dc0191b with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c2c99be4-d060-440f-a043-4964e48a2367_isrForCrosstalkSources_1031141_39.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c2c99be4-d060-440f-a043-4964e48a2367_isrForCrosstalkSources_1031141_39.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3023d72762fbe597a2f441c48d6845b with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1ff4f013-5655-4e6c-8d0d-35cd5c721f37_isrForCrosstalkSources_1031140_32.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1ff4f013-5655-4e6c-8d0d-35cd5c721f37_isrForCrosstalkSources_1031140_32.stderr 2024-03-17 01:27:23 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local 2024-03-17 01:27:23 proc_lsst.multi:146 [INFO] found job 17142638 in provider astro proc_lsst.multi INFO: found job 17142638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:23 proc_lsst.multi:146 [INFO] found job 17142639 in provider astro proc_lsst.multi INFO: found job 17142639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:24 proc_lsst.multi:146 [INFO] found job 17142640 in provider astro proc_lsst.multi INFO: found job 17142640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:25 proc_lsst.multi:146 [INFO] found job 17142641 in provider astro proc_lsst.multi INFO: found job 17142641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:25 proc_lsst.multi:146 [INFO] found job 17142642 in provider astro proc_lsst.multi INFO: found job 17142642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:25 proc_lsst.multi:146 [INFO] found job 17142643 in provider astro proc_lsst.multi INFO: found job 17142643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:25 proc_lsst.multi:146 [INFO] found job 17142644 in provider astro proc_lsst.multi INFO: found job 17142644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 360362025e6f88739b1c2d63654189e4 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1c4905dd-ece5-4433-b58e-6b1fef2ed503_isrForCrosstalkSources_1031141_24.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/1c4905dd-ece5-4433-b58e-6b1fef2ed503_isrForCrosstalkSources_1031141_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:26 proc_lsst.multi:146 [INFO] found job 17142645 in provider astro proc_lsst.multi INFO: found job 17142645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:26 proc_lsst.multi:146 [INFO] found job 17142646 in provider astro proc_lsst.multi INFO: found job 17142646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:26 proc_lsst.multi:146 [INFO] found job 17142647 in provider astro proc_lsst.multi INFO: found job 17142647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:27 proc_lsst.multi:146 [INFO] found job 17142648 in provider astro proc_lsst.multi INFO: found job 17142648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:27 proc_lsst.multi:146 [INFO] found job 17142650 in provider astro proc_lsst.multi INFO: found job 17142650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4326239a675cced532a599e2f852db40 with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49b7dfb3-09de-443a-8ec2-54e04297f2c4_isrForCrosstalkSources_1031139_30.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49b7dfb3-09de-443a-8ec2-54e04297f2c4_isrForCrosstalkSources_1031139_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:27 proc_lsst.multi:146 [INFO] found job 17142651 in provider astro proc_lsst.multi INFO: found job 17142651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:28 proc_lsst.multi:146 [INFO] found job 17142652 in provider astro proc_lsst.multi INFO: found job 17142652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:28 proc_lsst.multi:146 [INFO] found job 17142653 in provider astro proc_lsst.multi INFO: found job 17142653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:29 proc_lsst.multi:146 [INFO] found job 17142654 in provider astro proc_lsst.multi INFO: found job 17142654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:29 proc_lsst.multi:146 [INFO] found job 17142655 in provider astro proc_lsst.multi INFO: found job 17142655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c629db5398696c53885a3ee134e92ed8 with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ee51e2fc-813f-404a-bd2f-118fe7542826_isrForCrosstalkSources_1031141_25.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ee51e2fc-813f-404a-bd2f-118fe7542826_isrForCrosstalkSources_1031141_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:30 proc_lsst.multi:146 [INFO] found job 17142656 in provider astro proc_lsst.multi INFO: found job 17142656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:30 proc_lsst.multi:146 [INFO] found job 17142657 in provider astro proc_lsst.multi INFO: found job 17142657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13e4da44a644adc02489abebac980482 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/35bc6bf7-6368-403b-a001-633a09c93cec_isrForCrosstalkSources_1031139_20.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/35bc6bf7-6368-403b-a001-633a09c93cec_isrForCrosstalkSources_1031139_20.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41f8f013937f0bba00d4ad2345c2f687 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fdfb1bea-7a7f-4a72-bbf1-87c177b82a80_isrForCrosstalkSources_1031139_16.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fdfb1bea-7a7f-4a72-bbf1-87c177b82a80_isrForCrosstalkSources_1031139_16.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff1a5e9eb7c180b51389b2c5d9b7de57 with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0c005b82-1bf5-4c20-accc-d75c5f14755a_isrForCrosstalkSources_1031139_46.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0c005b82-1bf5-4c20-accc-d75c5f14755a_isrForCrosstalkSources_1031139_46.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc586df296472e20a393b588563494fb with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/97d0bc63-00fb-43a7-8622-921a1f94d70c_isrForCrosstalkSources_1031139_44.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/97d0bc63-00fb-43a7-8622-921a1f94d70c_isrForCrosstalkSources_1031139_44.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5031d9365589d04ed2a4c06d3e7315a4 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/91de433f-6efc-4616-b537-eb51789bd39a_isrForCrosstalkSources_1031141_46.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/91de433f-6efc-4616-b537-eb51789bd39a_isrForCrosstalkSources_1031141_46.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8773183d29d5bed59edb4f68da87a67 with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f9587144-2a60-48ea-9019-bb66f72896d0_isrForCrosstalkSources_1031141_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f9587144-2a60-48ea-9019-bb66f72896d0_isrForCrosstalkSources_1031141_32.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 155ee2e3d19154bdeb659cecb850bb5d with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f3ab147e-0dd2-454c-b011-5c20796557d8_isrForCrosstalkSources_1031140_12.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f3ab147e-0dd2-454c-b011-5c20796557d8_isrForCrosstalkSources_1031140_12.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba5805c17b2ddf6cb4bf2d07b2e5f881 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d386a25c-af89-467b-a7d8-924b1ba4e66c_isrForCrosstalkSources_1031139_27.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d386a25c-af89-467b-a7d8-924b1ba4e66c_isrForCrosstalkSources_1031139_27.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86890fcc0e3f4a5e1722c50994ddd920 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/3a5382b9-2a89-45ba-bf07-99faf834fc8e_isrForCrosstalkSources_1031140_44.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/3a5382b9-2a89-45ba-bf07-99faf834fc8e_isrForCrosstalkSources_1031140_44.stderr parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b46cb030b10f7502ce190edd897b805 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/777d4ca4-fcd8-46a7-9cd4-e35e64771508_isrForCrosstalkSources_1031140_57.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/777d4ca4-fcd8-46a7-9cd4-e35e64771508_isrForCrosstalkSources_1031140_57.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry baf060c055d34bba0befdf6ad0b48e43 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ddca9636-042c-4e30-8e4e-59e9a57472a8_isrForCrosstalkSources_1031141_10.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/ddca9636-042c-4e30-8e4e-59e9a57472a8_isrForCrosstalkSources_1031141_10.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51baa358513eed010c66d49b77cc67ea with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/11ad3842-4314-46ee-8652-efc4381814fb_isrForCrosstalkSources_1031139_53.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/11ad3842-4314-46ee-8652-efc4381814fb_isrForCrosstalkSources_1031139_53.stderr parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d978723d1ac45d6d2bcf3eef34e25745 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fb3227df-186b-4185-8af1-22b3eb201a0d_isrForCrosstalkSources_1031139_37.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/fb3227df-186b-4185-8af1-22b3eb201a0d_isrForCrosstalkSources_1031139_37.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c24a485cf2d7c7740655c1b81c7b6e81 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bcbf4c3d-463f-48af-bd0b-b86da68182be_isrForCrosstalkSources_1031140_14.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bcbf4c3d-463f-48af-bd0b-b86da68182be_isrForCrosstalkSources_1031140_14.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f3aa4e46422d1091d3e6d13f8dc5493 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/64acc5b2-7a1a-4578-a200-0cd5222c2e84_isrForCrosstalkSources_1031141_55.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/64acc5b2-7a1a-4578-a200-0cd5222c2e84_isrForCrosstalkSources_1031141_55.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdeb498008004d7993cf6dc7dc4920f with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/547c3dda-6383-452a-98fc-8707398e7f0b_isrForCrosstalkSources_1031139_15.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/547c3dda-6383-452a-98fc-8707398e7f0b_isrForCrosstalkSources_1031139_15.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e1ce25357769c2a32a2504c7a6c9942 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/12177999-83e5-4dac-a759-f97fcb41bcf8_isrForCrosstalkSources_1031141_16.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/12177999-83e5-4dac-a759-f97fcb41bcf8_isrForCrosstalkSources_1031141_16.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2708e198fa982e1e38c165e1ebca0c3b with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6dfbd6cf-8a7c-44d7-8a5f-893c0db0af92_isrForCrosstalkSources_1031139_1.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6dfbd6cf-8a7c-44d7-8a5f-893c0db0af92_isrForCrosstalkSources_1031139_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:31 proc_lsst.multi:146 [INFO] found job 17142658 in provider astro proc_lsst.multi INFO: found job 17142658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:32 proc_lsst.multi:146 [INFO] found job 17142659 in provider astro proc_lsst.multi INFO: found job 17142659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca2a0b118869e0e8bb38ffed6bf689e4 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b67671b5-82d1-4ff1-ab3b-c973ba34d4f2_isrForCrosstalkSources_1031140_33.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/b67671b5-82d1-4ff1-ab3b-c973ba34d4f2_isrForCrosstalkSources_1031140_33.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b93952c333f42cd3471dacdd7f94e034 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/7ed409fd-0d32-495d-8276-3c2215d2c3b0_isrForCrosstalkSources_1031140_26.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/7ed409fd-0d32-495d-8276-3c2215d2c3b0_isrForCrosstalkSources_1031140_26.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e19b68e9d4bfa554d76762bf1bfeb257 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/01575a4a-1a42-469d-b490-e1d84cd591e3_isrForCrosstalkSources_1031139_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/01575a4a-1a42-469d-b490-e1d84cd591e3_isrForCrosstalkSources_1031139_7.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dcb68b40f192064c4cc52f5d0677b2d6 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/73c2c216-063a-48ff-a6b2-29cb09e882a5_isrForCrosstalkSources_1031139_38.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/73c2c216-063a-48ff-a6b2-29cb09e882a5_isrForCrosstalkSources_1031139_38.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffb7a7722f84f6fcb39876ccab19486b with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/2ff2d47a-10e1-49af-88e5-25f8ecbfa131_isrForCrosstalkSources_1031139_36.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/2ff2d47a-10e1-49af-88e5-25f8ecbfa131_isrForCrosstalkSources_1031139_36.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3a2465a78c6dac1c079a1f348caad37 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b2dc2415-1d4b-4097-8594-9b762c2501e6_isrForCrosstalkSources_1031139_3.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b2dc2415-1d4b-4097-8594-9b762c2501e6_isrForCrosstalkSources_1031139_3.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5371705f565198e3a529babf4f2033b3 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/95717a0d-1523-4a7e-8642-b01669811496_isrForCrosstalkSources_1031139_11.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/95717a0d-1523-4a7e-8642-b01669811496_isrForCrosstalkSources_1031139_11.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80497456bfa3d0f07138df17b52de8cf with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4960920d-6960-48df-9be2-4e2c20daf1c6_isrForCrosstalkSources_1031139_50.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/4960920d-6960-48df-9be2-4e2c20daf1c6_isrForCrosstalkSources_1031139_50.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b20cb2f4065a6890ed8267cdbf3b2a with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/23293ee9-6c24-4a9d-91a4-d0a3e026bdcb_isrForCrosstalkSources_1031139_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/23293ee9-6c24-4a9d-91a4-d0a3e026bdcb_isrForCrosstalkSources_1031139_42.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b05a72341357aeae951ce0e29cc1db03 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/735db864-6318-4ed9-bf96-1b34c6b1f9de_isrForCrosstalkSources_1031140_4.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/735db864-6318-4ed9-bf96-1b34c6b1f9de_isrForCrosstalkSources_1031140_4.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 416ff3968d17f20854380aa1ca60169f with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1d35a7bc-9b04-4a68-9657-e50ff39bf387_isrForCrosstalkSources_1031140_23.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1d35a7bc-9b04-4a68-9657-e50ff39bf387_isrForCrosstalkSources_1031140_23.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bd1c1fa4b67c75c03bc17b796c18381 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/962b8357-6872-46c3-9e83-6f1f4d652e94_isrForCrosstalkSources_1031140_45.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/962b8357-6872-46c3-9e83-6f1f4d652e94_isrForCrosstalkSources_1031140_45.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 133710fd2d36914024e286fbbba0209d with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/108ac57c-8cf5-4002-a450-b930e00a52a1_isrForCrosstalkSources_1031141_52.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/108ac57c-8cf5-4002-a450-b930e00a52a1_isrForCrosstalkSources_1031141_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:33 proc_lsst.multi:146 [INFO] found job 17142660 in provider astro proc_lsst.multi INFO: found job 17142660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:34 proc_lsst.multi:146 [INFO] found job 17142661 in provider astro proc_lsst.multi INFO: found job 17142661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09e5bb3d3cdb05b5dfd623933f6f867c with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6d779be7-93f0-43e3-b888-5159c3232f93_isrForCrosstalkSources_1031140_53.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6d779be7-93f0-43e3-b888-5159c3232f93_isrForCrosstalkSources_1031140_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:34 proc_lsst.multi:146 [INFO] found job 17142662 in provider astro proc_lsst.multi INFO: found job 17142662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04f5aa34534fb7102c1e994f83a03931 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5fff2fd1-6936-4be6-a98d-5443495d697d_isrForCrosstalkSources_1031141_47.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5fff2fd1-6936-4be6-a98d-5443495d697d_isrForCrosstalkSources_1031141_47.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2463b65e452449c2736e23e0be8eae2b with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5b602c36-a7dc-4d3e-ae06-ac850aa978ac_isrForCrosstalkSources_1031140_48.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5b602c36-a7dc-4d3e-ae06-ac850aa978ac_isrForCrosstalkSources_1031140_48.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b18fe8d668142f2d84851664abe5b8a9 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f807edda-e310-48d9-82c9-40ae011875b8_isrForCrosstalkSources_1031141_49.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/f807edda-e310-48d9-82c9-40ae011875b8_isrForCrosstalkSources_1031141_49.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66d04356283d04bc0fb11d488a5a5174 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0bdf451e-68ee-4cb8-80c2-1c7a5569e506_isrForCrosstalkSources_1031141_57.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0bdf451e-68ee-4cb8-80c2-1c7a5569e506_isrForCrosstalkSources_1031141_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:35 proc_lsst.multi:146 [INFO] found job 17142663 in provider astro proc_lsst.multi INFO: found job 17142663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:35 proc_lsst.multi:146 [INFO] found job 17142664 in provider astro proc_lsst.multi INFO: found job 17142664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:35 proc_lsst.multi:146 [INFO] found job 17142665 in provider astro proc_lsst.multi INFO: found job 17142665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d89336cbf0c46cd0820c54194d2d75e with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/163b163a-a213-407a-87c4-26e25d189301_isrForCrosstalkSources_1031140_49.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/163b163a-a213-407a-87c4-26e25d189301_isrForCrosstalkSources_1031140_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:36 proc_lsst.multi:146 [INFO] found job 17142666 in provider astro proc_lsst.multi INFO: found job 17142666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8d7ac4fd5b35aea4782180a0ab7f093 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/459745a3-1be8-4f17-82f8-2a23d761bc0b_isrForCrosstalkSources_1031140_11.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/459745a3-1be8-4f17-82f8-2a23d761bc0b_isrForCrosstalkSources_1031140_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:36 proc_lsst.multi:146 [INFO] found job 17142667 in provider astro proc_lsst.multi INFO: found job 17142667 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 881f4dbcf5ca042237a288c411ace957 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3d3343df-6edf-4921-ab0a-7654357d5921_isrForCrosstalkSources_1031139_10.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3d3343df-6edf-4921-ab0a-7654357d5921_isrForCrosstalkSources_1031139_10.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 258c62db79699572cc6d682862d08d6a with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/9098554f-88a6-4309-869d-7d82e5650dda_isrForCrosstalkSources_1031139_43.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/9098554f-88a6-4309-869d-7d82e5650dda_isrForCrosstalkSources_1031139_43.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 57b686bd0877e1edc4ddd274c3bee328 with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/86635514-ec59-4a5e-bc56-e3cb4f11152e_isrForCrosstalkSources_1031140_20.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/86635514-ec59-4a5e-bc56-e3cb4f11152e_isrForCrosstalkSources_1031140_20.stderr parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 743e7caba99f5a0b326a88b1fd02363d with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/292d0d9c-99a2-4bc2-bd4e-22dcfc855429_isrForCrosstalkSources_1031141_33.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/292d0d9c-99a2-4bc2-bd4e-22dcfc855429_isrForCrosstalkSources_1031141_33.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ac77a177fd05cb37d9a6772121b9c7f with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b229ab16-45fa-460c-8288-6d093ecce3c2_isrForCrosstalkSources_1031141_15.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b229ab16-45fa-460c-8288-6d093ecce3c2_isrForCrosstalkSources_1031141_15.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67750d22ed9d6d8458382a09cdd12c2d with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/645fe08d-3297-483e-b3f3-2a74fa4b4843_isrForCrosstalkSources_1031141_6.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/645fe08d-3297-483e-b3f3-2a74fa4b4843_isrForCrosstalkSources_1031141_6.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5178ed81ec4006ab345cd0b2c8476925 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/978110b6-8dca-444b-a506-ca8e24e09bcd_isrForCrosstalkSources_1031140_6.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/978110b6-8dca-444b-a506-ca8e24e09bcd_isrForCrosstalkSources_1031140_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:37 proc_lsst.multi:146 [INFO] found job 17142668 in provider astro proc_lsst.multi INFO: found job 17142668 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:37 proc_lsst.multi:146 [INFO] found job 17142669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:37 proc_lsst.multi:146 [INFO] found job 17142670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:38 proc_lsst.multi:146 [INFO] found job 17142671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:38 proc_lsst.multi:146 [INFO] found job 17142672 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142672 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f55a3e1f0f0f45485f794c12c2f9912f with result from task 83 parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:38 proc_lsst.multi:146 [INFO] found job 17142673 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142673 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/973f6170-553c-47ad-88fd-371d0067294a_isrForCrosstalkSources_1031141_13.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/973f6170-553c-47ad-88fd-371d0067294a_isrForCrosstalkSources_1031141_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:38 proc_lsst.multi:146 [INFO] found job 17142674 in provider escience_a40 proc_lsst.multi INFO: found job 17142674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:39 proc_lsst.multi:146 [INFO] found job 17142675 in provider escience_a40 proc_lsst.multi INFO: found job 17142675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:39 proc_lsst.multi:146 [INFO] found job 17142676 in provider escience_a40 proc_lsst.multi INFO: found job 17142676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:40 proc_lsst.multi:146 [INFO] found job 17142677 in provider escience_a40 proc_lsst.multi INFO: found job 17142677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ebda810009fd788998bfb240087a562 with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d5258297-c0a8-4aad-9c86-16ea9cbcef3d_isrForCrosstalkSources_1031139_51.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d5258297-c0a8-4aad-9c86-16ea9cbcef3d_isrForCrosstalkSources_1031139_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:40 proc_lsst.multi:146 [INFO] found job 17142678 in provider escience_a40 proc_lsst.multi INFO: found job 17142678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:40 proc_lsst.multi:146 [INFO] found job 17142683 in provider escience_a40 proc_lsst.multi INFO: found job 17142683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:41 proc_lsst.multi:146 [INFO] found job 17142687 in provider escience_a40 proc_lsst.multi INFO: found job 17142687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:41 proc_lsst.multi:146 [INFO] found job 17142688 in provider escience_a40 proc_lsst.multi INFO: found job 17142688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:42 proc_lsst.multi:146 [INFO] found job 17142689 in provider escience_a40 proc_lsst.multi INFO: found job 17142689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:42 proc_lsst.multi:146 [INFO] found job 17142690 in provider escience_a40 proc_lsst.multi INFO: found job 17142690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85b382f02c61a5cb3583df72983702d1 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eb7311c-b830-4a89-8cbd-283587fe6eca_isrForCrosstalkSources_1031140_58.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eb7311c-b830-4a89-8cbd-283587fe6eca_isrForCrosstalkSources_1031140_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 100 parsl.dataflow.strategy DEBUG: Executor multi has 100 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 100 parsl.dataflow.strategy DEBUG: Executor multi has 100 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d8e4cd4bf40a83dec6838a880a389ff with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/411fdfe4-c8a6-415e-8259-437740463842_isrForCrosstalkSources_1031140_15.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/411fdfe4-c8a6-415e-8259-437740463842_isrForCrosstalkSources_1031140_15.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fbc17f44fdd8d6166ffa76157e189e0 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6cd6ab74-ce11-434b-a975-08a8cbc182d1_isrForCrosstalkSources_1031140_60.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/6cd6ab74-ce11-434b-a975-08a8cbc182d1_isrForCrosstalkSources_1031140_60.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a188d785172eb27f82cef3c44c373f5 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4d633103-52ad-4649-ab44-5268babfe8d3_isrForCrosstalkSources_1031141_22.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4d633103-52ad-4649-ab44-5268babfe8d3_isrForCrosstalkSources_1031141_22.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e00fa37582bff54485d988c405da2c55 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/52585796-127a-413a-9a0e-523bcf34234a_isrForCrosstalkSources_1031140_41.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/52585796-127a-413a-9a0e-523bcf34234a_isrForCrosstalkSources_1031140_41.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7cd5751ac4583f429a6fa2261d678990 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b43bc7a1-f41c-4f95-9bfa-fc9f1d255649_isrForCrosstalkSources_1031141_11.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b43bc7a1-f41c-4f95-9bfa-fc9f1d255649_isrForCrosstalkSources_1031141_11.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 410d2118c89df913fd43ab7a2a1fc7ad with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e9ab3015-ceaa-4371-b36e-44c84d77c352_isrForCrosstalkSources_1031141_44.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e9ab3015-ceaa-4371-b36e-44c84d77c352_isrForCrosstalkSources_1031141_44.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 285de97bd2face2fff0884609406892f with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/72db7b6d-5ba9-4546-9099-a1258147f8ea_isrForCrosstalkSources_1031139_34.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/72db7b6d-5ba9-4546-9099-a1258147f8ea_isrForCrosstalkSources_1031139_34.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b8a416147c10ae50a28bcfb97da0a8f with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/f235b0f3-cb7a-4a39-8170-3219da353b14_isrForCrosstalkSources_1031139_8.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/f235b0f3-cb7a-4a39-8170-3219da353b14_isrForCrosstalkSources_1031139_8.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee25e84761a9bd450c564a971a02052 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/83db9937-2b16-4ac1-b409-5de6c47e4d00_isrForCrosstalkSources_1031139_14.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/83db9937-2b16-4ac1-b409-5de6c47e4d00_isrForCrosstalkSources_1031139_14.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2fc9098cae190b3050e6dd461587dc5 with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/30a4cbf4-3aab-4ba4-92fc-91aca718d001_isrForCrosstalkSources_1031140_50.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/30a4cbf4-3aab-4ba4-92fc-91aca718d001_isrForCrosstalkSources_1031140_50.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81daaa6c2eee3f457e813b10a0e749f9 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/118c8a93-1097-4551-a897-5a33e8fb08a2_isrForCrosstalkSources_1031139_39.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/118c8a93-1097-4551-a897-5a33e8fb08a2_isrForCrosstalkSources_1031139_39.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2d0628a07966fd54dd6d73d0f80062f with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8ab713-fb8e-4fb3-968b-0a3ad6055783_isrForCrosstalkSources_1031139_41.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/aa8ab713-fb8e-4fb3-968b-0a3ad6055783_isrForCrosstalkSources_1031139_41.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b87160d6404576a1421d167d00310d3 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/99eb7a64-db07-41e2-8cfe-5f4c749dc1c1_isrForCrosstalkSources_1031141_26.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/99eb7a64-db07-41e2-8cfe-5f4c749dc1c1_isrForCrosstalkSources_1031141_26.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8efc5578c49fb88554634def7d283e2f with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e2e6e8f0-e547-4f51-8abd-42c45d41a03c_isrForCrosstalkSources_1031141_20.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e2e6e8f0-e547-4f51-8abd-42c45d41a03c_isrForCrosstalkSources_1031141_20.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7222e68c39104efb57275c7f8ff38385 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/89c7ea18-7040-4d11-b9ba-e992dd58c469_isrForCrosstalkSources_1031139_28.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/89c7ea18-7040-4d11-b9ba-e992dd58c469_isrForCrosstalkSources_1031139_28.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311e5276bec1666a5c243c0eba9e7296 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bdd6906f-e50a-460f-82f9-495c4c7af655_isrForCrosstalkSources_1031140_25.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bdd6906f-e50a-460f-82f9-495c4c7af655_isrForCrosstalkSources_1031140_25.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f887fcda17177f82a80cab8bf4e59f17 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61ff1abb-136e-44b1-91e3-48553c96ad49_isrForCrosstalkSources_1031140_34.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61ff1abb-136e-44b1-91e3-48553c96ad49_isrForCrosstalkSources_1031140_34.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6072c44a0972138491b9dfc22319e2e5 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8de2a4b1-2bf5-43ba-9f96-7261bb592b49_isrForCrosstalkSources_1031139_49.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/8de2a4b1-2bf5-43ba-9f96-7261bb592b49_isrForCrosstalkSources_1031139_49.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d628b64473bd62bc92271828b3da90ca with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/14f2c932-c2af-4116-be04-c2421dc7948a_isrForCrosstalkSources_1031139_57.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/14f2c932-c2af-4116-be04-c2421dc7948a_isrForCrosstalkSources_1031139_57.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7728a6857460abfc000f462e417ea3a4 with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/30ef094e-3743-4f93-9ba4-461d830abc7e_isrForCrosstalkSources_1031139_54.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/30ef094e-3743-4f93-9ba4-461d830abc7e_isrForCrosstalkSources_1031139_54.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d94b598ab9ba392806c97bbc0ef7b3 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7391efe6-19f8-4f66-84d1-e3d8808c71aa_isrForCrosstalkSources_1031141_17.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7391efe6-19f8-4f66-84d1-e3d8808c71aa_isrForCrosstalkSources_1031141_17.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77ca0a48a79f0da421fc08d4d420e966 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/16b927a1-b2bd-4a99-be2c-07e0c06ec22c_isrForCrosstalkSources_1031140_39.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/16b927a1-b2bd-4a99-be2c-07e0c06ec22c_isrForCrosstalkSources_1031140_39.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee38206f5578ab7946d1a122bd118e5e with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/bd63a19c-d087-4d30-acb4-d685e3561949_isrForCrosstalkSources_1031139_40.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/bd63a19c-d087-4d30-acb4-d685e3561949_isrForCrosstalkSources_1031139_40.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2faa4a5d58f8da9a859d49654157005a with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b81f6d17-3a04-46b5-a3af-64f519f47938_isrForCrosstalkSources_1031141_4.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/b81f6d17-3a04-46b5-a3af-64f519f47938_isrForCrosstalkSources_1031141_4.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9f2634f54f96e9a62b76b5a57417ecc with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/63068b92-647c-4eb3-be56-e0d816a19732_isrForCrosstalkSources_1031140_40.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/63068b92-647c-4eb3-be56-e0d816a19732_isrForCrosstalkSources_1031140_40.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 569f7708ea3bf15339fe1a2f209fc03b with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5caec8f7-ee68-4689-aeec-a83b0fe04688_isrForCrosstalkSources_1031141_40.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5caec8f7-ee68-4689-aeec-a83b0fe04688_isrForCrosstalkSources_1031141_40.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96acb820cd503645a31d98846211c41 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d3d05bb5-69c8-48d4-abab-1e81862a9420_isrForCrosstalkSources_1031139_56.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/d3d05bb5-69c8-48d4-abab-1e81862a9420_isrForCrosstalkSources_1031139_56.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1fdb9acb20d717535dde157496e0394 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/be30e73d-8c30-4be6-908b-059d46eea585_isrForCrosstalkSources_1031141_1.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/be30e73d-8c30-4be6-908b-059d46eea585_isrForCrosstalkSources_1031141_1.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c353f86e8fc98a52129c461fbf3d348 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/9e6a074b-933f-41d2-8810-d4563bc0e841_isrForCrosstalkSources_1031141_62.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/9e6a074b-933f-41d2-8810-d4563bc0e841_isrForCrosstalkSources_1031141_62.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9855d12c0592177474a761e770be41c6 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6be435ee-8066-4c1b-9cf9-8c35e036d5f5_isrForCrosstalkSources_1031139_31.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/6be435ee-8066-4c1b-9cf9-8c35e036d5f5_isrForCrosstalkSources_1031139_31.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac91d8d0e9b83c2abd8bb609384327c3 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/088ca952-139a-41b4-a811-98fffc32ed57_isrForCrosstalkSources_1031139_2.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/088ca952-139a-41b4-a811-98fffc32ed57_isrForCrosstalkSources_1031139_2.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce13ff131721be6d83ea33598b6a4c63 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/43391194-1001-4b59-b548-3ef473d97837_isrForCrosstalkSources_1031139_55.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/43391194-1001-4b59-b548-3ef473d97837_isrForCrosstalkSources_1031139_55.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5d04218b5931209a8fcd7fe49eea31 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/85ba50be-b139-481b-bdb9-31041f8765e6_isrForCrosstalkSources_1031140_61.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/85ba50be-b139-481b-bdb9-31041f8765e6_isrForCrosstalkSources_1031140_61.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30e9a4ecfb911c08b664ae0e101da17d with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/fc899088-9444-43f4-be2d-bfb3640f993f_isrForCrosstalkSources_1031141_28.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/fc899088-9444-43f4-be2d-bfb3640f993f_isrForCrosstalkSources_1031141_28.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce5a167313862ce9c523f3f75b6d808a with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/3819be19-b762-4fb1-a8d2-092a59bb958e_isrForCrosstalkSources_1031141_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/3819be19-b762-4fb1-a8d2-092a59bb958e_isrForCrosstalkSources_1031141_42.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd5ae84eef423d7243392a12df66c95c with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/dd40c1e4-6ada-4e51-b4b8-d8ccfb9d7892_isrForCrosstalkSources_1031141_56.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/dd40c1e4-6ada-4e51-b4b8-d8ccfb9d7892_isrForCrosstalkSources_1031141_56.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91ef060d27554f11e34e627557b44ba5 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/dd1eb700-af67-424d-8457-12c93c06f6a0_isrForCrosstalkSources_1031140_56.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/dd1eb700-af67-424d-8457-12c93c06f6a0_isrForCrosstalkSources_1031140_56.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d631dc1c87b096d03844da6c194dbda with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/df83a869-383a-409a-ad2c-3383a6d49040_isrForCrosstalkSources_1031141_19.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/df83a869-383a-409a-ad2c-3383a6d49040_isrForCrosstalkSources_1031141_19.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98c595c92c517ab7f4cb8d7f1b2e6887 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/caff82c9-72f9-4c98-a45d-d20c9c67b455_isrForCrosstalkSources_1031141_45.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/caff82c9-72f9-4c98-a45d-d20c9c67b455_isrForCrosstalkSources_1031141_45.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13097a8a529ceda7a5e017ce2e5cab68 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/284569b5-c39f-4735-8dae-6d58d735939f_isrForCrosstalkSources_1031141_34.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/284569b5-c39f-4735-8dae-6d58d735939f_isrForCrosstalkSources_1031141_34.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 60 parsl.dataflow.strategy DEBUG: Executor multi has 60 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a71577cb98a238bd974551aaa8c7349 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/32f7b5c2-76a4-4379-ad96-08fce246a775_isrForCrosstalkSources_1031141_38.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/32f7b5c2-76a4-4379-ad96-08fce246a775_isrForCrosstalkSources_1031141_38.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c54b9680343284d5793b415a4a6916a with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/026ff445-2e8a-468c-91ad-b26b1fdd8af4_isrForCrosstalkSources_1031140_10.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/026ff445-2e8a-468c-91ad-b26b1fdd8af4_isrForCrosstalkSources_1031140_10.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 993f046211bb1697413fc2b488c863d0 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bbda9e4d-881b-4b1f-b3c6-29c6ed16c225_isrForCrosstalkSources_1031140_24.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bbda9e4d-881b-4b1f-b3c6-29c6ed16c225_isrForCrosstalkSources_1031140_24.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f08c496469af79543bb7652a8d7b4900 with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d6fe5f50-5b02-4c9e-bba0-d56a358c5223_isrForCrosstalkSources_1031141_53.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d6fe5f50-5b02-4c9e-bba0-d56a358c5223_isrForCrosstalkSources_1031141_53.stderr parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b99c1602c70a55b8452c7ab825f4a54f with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/009a647e-405e-4d12-a63c-16fbfd6b18c0_isrForCrosstalkSources_1031141_60.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/009a647e-405e-4d12-a63c-16fbfd6b18c0_isrForCrosstalkSources_1031141_60.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fee5d2701cd4d56a1634ee679a91c75f with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/92edf746-a6f8-48b3-9136-23a002d62398_isrForCrosstalkSources_1031140_29.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/92edf746-a6f8-48b3-9136-23a002d62398_isrForCrosstalkSources_1031140_29.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d0438278c3d06569efa142810eed45c with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/bff59b81-b99d-4fd7-9579-151f8be10e93_isrForCrosstalkSources_1031141_23.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/bff59b81-b99d-4fd7-9579-151f8be10e93_isrForCrosstalkSources_1031141_23.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72895f26435f65587fe5f8e9d087cdfa with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3a3f072f-6c30-4e5b-86a0-ad897f67f278_isrForCrosstalkSources_1031139_9.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3a3f072f-6c30-4e5b-86a0-ad897f67f278_isrForCrosstalkSources_1031139_9.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 52 parsl.dataflow.strategy DEBUG: Executor multi has 52 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b4bddd637b52aedf32cc81d4582a6b with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/1e38b595-244f-48c2-bdc0-2ca807099f3f_isrForCrosstalkSources_1031139_25.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/1e38b595-244f-48c2-bdc0-2ca807099f3f_isrForCrosstalkSources_1031139_25.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86092eb9888d72a8b9049ae60f84a1f3 with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/25bfa358-c0a6-4ba7-9123-1c0fcce4521f_isrForCrosstalkSources_1031139_19.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/25bfa358-c0a6-4ba7-9123-1c0fcce4521f_isrForCrosstalkSources_1031139_19.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33761f6d614d004ac81b3cdbc037e9e5 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0230f5c6-af37-4a03-83ab-52bf7eea8f59_isrForCrosstalkSources_1031139_23.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/0230f5c6-af37-4a03-83ab-52bf7eea8f59_isrForCrosstalkSources_1031139_23.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d04bbc4b4d003eba830101583aabb52 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b27c7c04-10e4-4646-98ca-a3538662ff13_isrForCrosstalkSources_1031139_32.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b27c7c04-10e4-4646-98ca-a3538662ff13_isrForCrosstalkSources_1031139_32.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e2c787eac0b78ba321a221ad76c35eb with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/08a54dba-8e81-45fb-916e-7a0ca2df499a_isrForCrosstalkSources_1031141_29.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/08a54dba-8e81-45fb-916e-7a0ca2df499a_isrForCrosstalkSources_1031141_29.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec86908268ab727388e8d1e66bf46f76 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/657eb189-78c6-4a33-aa7d-272a685003bd_isrForCrosstalkSources_1031139_61.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/657eb189-78c6-4a33-aa7d-272a685003bd_isrForCrosstalkSources_1031139_61.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a43197bff1b57a7d99ee994acf6f4597 with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6b55a1fd-50c9-444e-b404-9a934bdc10e5_isrForCrosstalkSources_1031141_30.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6b55a1fd-50c9-444e-b404-9a934bdc10e5_isrForCrosstalkSources_1031141_30.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d4ebcdba2a50b18b825e6ae696ebbe5 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/31f99e87-724f-4bf6-889c-128845b23fdd_isrForCrosstalkSources_1031139_13.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/31f99e87-724f-4bf6-889c-128845b23fdd_isrForCrosstalkSources_1031139_13.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7617cc59f74d0de84498e2d15dfaa04f with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/5fbce5d9-0de2-4184-a057-4abe39ae2f5d_isrForCrosstalkSources_1031139_47.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/5fbce5d9-0de2-4184-a057-4abe39ae2f5d_isrForCrosstalkSources_1031139_47.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ed20652857db26455f597240574bdf91 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a3141917-be51-4a8b-8045-0c3cca74fcf3_isrForCrosstalkSources_1031140_54.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a3141917-be51-4a8b-8045-0c3cca74fcf3_isrForCrosstalkSources_1031140_54.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a398627558af4931047700db1b5eb19f with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/89466e0d-819a-4ce0-9cff-bd7ecaae0c44_isrForCrosstalkSources_1031140_2.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/89466e0d-819a-4ce0-9cff-bd7ecaae0c44_isrForCrosstalkSources_1031140_2.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0639441bb1616bbcb92a4ceff5dd9c9 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/481a3d0f-2a3d-459a-8f06-7d19e5222eb0_isrForCrosstalkSources_1031140_52.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/481a3d0f-2a3d-459a-8f06-7d19e5222eb0_isrForCrosstalkSources_1031140_52.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49094232bc1ebbe0f998483956ea61cd with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2ce7d325-0fc5-4697-b84b-3cb06e6534b0_isrForCrosstalkSources_1031140_43.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/2ce7d325-0fc5-4697-b84b-3cb06e6534b0_isrForCrosstalkSources_1031140_43.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 509f6315278effa9d57767a062975742 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8fc56329-3b40-4518-a90d-3cce51d31ed7_isrForCrosstalkSources_1031141_27.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/8fc56329-3b40-4518-a90d-3cce51d31ed7_isrForCrosstalkSources_1031141_27.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de2e79a6f6c30e546affea3f02f92e15 with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/330c310d-979c-4d0e-9ebd-1c7e13325555_isrForCrosstalkSources_1031140_21.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/330c310d-979c-4d0e-9ebd-1c7e13325555_isrForCrosstalkSources_1031140_21.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44651564911353e643a2b94e0f30cfca with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/390d8cbc-21b9-4cfd-826a-617982fc0593_isrForCrosstalkSources_1031139_60.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/390d8cbc-21b9-4cfd-826a-617982fc0593_isrForCrosstalkSources_1031139_60.stderr 2024-03-17 01:27:58 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 890d51aec8f7d43a419b35431e43974c with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/ae165f66-9705-4362-9e3a-7cab76c79c7d_isrForCrosstalkSources_1031140_17.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/ae165f66-9705-4362-9e3a-7cab76c79c7d_isrForCrosstalkSources_1031140_17.stderr 2024-03-17 01:27:58 proc_lsst.multi:146 [INFO] found job 17142638 in provider astro proc_lsst.multi INFO: found job 17142638 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119202ed25eac9e23bc69ed9ff8c1b64 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fbb7a654-6be2-4a3a-a137-7686c31d39a8_isrForCrosstalkSources_1031140_59.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/fbb7a654-6be2-4a3a-a137-7686c31d39a8_isrForCrosstalkSources_1031140_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:58 proc_lsst.multi:146 [INFO] found job 17142639 in provider astro proc_lsst.multi INFO: found job 17142639 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:59 proc_lsst.multi:146 [INFO] found job 17142640 in provider astro proc_lsst.multi INFO: found job 17142640 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a3a297eb7cd23d41e364b242a051aa3 with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61c62b4a-5eac-4404-8ae5-c57d7cb3d988_isrForCrosstalkSources_1031140_38.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/61c62b4a-5eac-4404-8ae5-c57d7cb3d988_isrForCrosstalkSources_1031140_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:27:59 proc_lsst.multi:146 [INFO] found job 17142641 in provider astro proc_lsst.multi INFO: found job 17142641 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bae4983cbbeac51ec162a7aef5ceccb6 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4dc8b663-05ae-474a-9926-1b1ad3512173_isrForCrosstalkSources_1031141_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/4dc8b663-05ae-474a-9926-1b1ad3512173_isrForCrosstalkSources_1031141_58.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e6b44ce3c619d01afd052028c12ffcc with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/a448fe8b-065f-45eb-9dd2-44256ff9a61e_isrForCrosstalkSources_1031141_8.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/a448fe8b-065f-45eb-9dd2-44256ff9a61e_isrForCrosstalkSources_1031141_8.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70f6f946db7c3d4c6e93c4bdce173424 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7817581e-4f75-4b15-ae9b-1dbddcc93f05_isrForCrosstalkSources_1031141_21.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/7817581e-4f75-4b15-ae9b-1dbddcc93f05_isrForCrosstalkSources_1031141_21.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1fc1405fd29ac2fbe76fdb2c15a1e27 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b7f01cdf-29d1-4d1b-b28f-05e9f3c246cb_isrForCrosstalkSources_1031139_17.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/b7f01cdf-29d1-4d1b-b28f-05e9f3c246cb_isrForCrosstalkSources_1031139_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:00 proc_lsst.multi:146 [INFO] found job 17142642 in provider astro proc_lsst.multi INFO: found job 17142642 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34c8f8334113724fc8548a411f5cf5e3 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a9f8c944-469c-40bf-8c26-dba4ffe66b7a_isrForCrosstalkSources_1031140_8.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/a9f8c944-469c-40bf-8c26-dba4ffe66b7a_isrForCrosstalkSources_1031140_8.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 741e7d0acf6882dfa600f4429c2e9b9c with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e14259c3-4e45-4674-98b7-5ab1d06341d0_isrForCrosstalkSources_1031141_14.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/e14259c3-4e45-4674-98b7-5ab1d06341d0_isrForCrosstalkSources_1031141_14.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad1731b17d3636849c56eb88f07fdf2e with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/123f03cd-b781-4362-b312-619008c6a691_isrForCrosstalkSources_1031141_18.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/123f03cd-b781-4362-b312-619008c6a691_isrForCrosstalkSources_1031141_18.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a261db90a70aefa827e52d074da6ebc7 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/77dc4262-1bdc-4dda-b7f4-7267e3f140c0_isrForCrosstalkSources_1031140_18.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/77dc4262-1bdc-4dda-b7f4-7267e3f140c0_isrForCrosstalkSources_1031140_18.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39a1b937d78e73660edbcb01863e20e6 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a3025387-8fd2-444b-9979-cd02df404d0f_isrForCrosstalkSources_1031139_18.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/a3025387-8fd2-444b-9979-cd02df404d0f_isrForCrosstalkSources_1031139_18.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1d148a20ea571990d8b3af09954a7903 with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c3f41142-061a-41c8-b303-70b29c5c8561_isrForCrosstalkSources_1031141_35.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c3f41142-061a-41c8-b303-70b29c5c8561_isrForCrosstalkSources_1031141_35.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8edda8e1eac56942bf6414347b825d6 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/657f9cd4-8e0e-42e2-b7a6-7227bf940a80_isrForCrosstalkSources_1031140_1.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/657f9cd4-8e0e-42e2-b7a6-7227bf940a80_isrForCrosstalkSources_1031140_1.stderr parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cffde0d27712b34bf46f88763375e72f with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/90539147-a963-409b-ab67-719377ca86ba_isrForCrosstalkSources_1031139_5.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/90539147-a963-409b-ab67-719377ca86ba_isrForCrosstalkSources_1031139_5.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 601c55ed7945c4f31a3f3d4ee4bb1945 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bb730d9b-399f-48e5-96dc-c54ee6fe1a3a_isrForCrosstalkSources_1031140_28.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/bb730d9b-399f-48e5-96dc-c54ee6fe1a3a_isrForCrosstalkSources_1031140_28.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a8fce1e0a2ccfae10fecb10d5892a28 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/d3e2cd45-fdd8-479c-acd1-7335c187a698_isrForCrosstalkSources_1031140_7.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/d3e2cd45-fdd8-479c-acd1-7335c187a698_isrForCrosstalkSources_1031140_7.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d04388390c56e2ef1cc613cce7d0810 with result from task 144 parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:00 proc_lsst.multi:146 [INFO] found job 17142643 in provider astro proc_lsst.multi INFO: found job 17142643 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8cd73728-c1af-4855-b954-66c69cee6eaf_isrForCrosstalkSources_1031140_35.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8cd73728-c1af-4855-b954-66c69cee6eaf_isrForCrosstalkSources_1031140_35.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e14e7bcd821f91a56da22b56f6e70c5 with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1a6683aa-9dd5-4672-9412-cb301cf9404f_isrForCrosstalkSources_1031140_31.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/1a6683aa-9dd5-4672-9412-cb301cf9404f_isrForCrosstalkSources_1031140_31.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f9692d4f358ad4dfe0f2f8e0832ae9e with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/31c2f8fb-38e9-4e3e-9fbf-47e89a472de2_isrForCrosstalkSources_1031140_5.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/31c2f8fb-38e9-4e3e-9fbf-47e89a472de2_isrForCrosstalkSources_1031140_5.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55905caa6198c51c257838a797164b0f with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5124a823-55ff-4292-a913-c05bf554b4de_isrForCrosstalkSources_1031141_43.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/5124a823-55ff-4292-a913-c05bf554b4de_isrForCrosstalkSources_1031141_43.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dad1d726405811d913b1da03dc829dc with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/819087f3-d146-4ec8-8971-69ca09b5093e_isrForCrosstalkSources_1031141_41.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/819087f3-d146-4ec8-8971-69ca09b5093e_isrForCrosstalkSources_1031141_41.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e58040a2ec90811346725539930ee410 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6cd59e70-2789-48cd-9c3e-a66ab931131b_isrForCrosstalkSources_1031141_50.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/6cd59e70-2789-48cd-9c3e-a66ab931131b_isrForCrosstalkSources_1031141_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:00 proc_lsst.multi:146 [INFO] found job 17142644 in provider astro proc_lsst.multi INFO: found job 17142644 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3f4ccbc4623015049fa65ddafe0bc99 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3e2fd1f3-f825-4423-bb79-9b1d28b23a72_isrForCrosstalkSources_1031139_58.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/3e2fd1f3-f825-4423-bb79-9b1d28b23a72_isrForCrosstalkSources_1031139_58.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e02bd194640a06e77f28bcc5552d5f0 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5f162c38-37d0-49b2-8b68-1b90d8acaa07_isrForCrosstalkSources_1031140_42.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/5f162c38-37d0-49b2-8b68-1b90d8acaa07_isrForCrosstalkSources_1031140_42.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 140c65b4735487e130d5a5a6959100d1 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0ea47cff-c54f-4111-b578-5e45eff8d333_isrForCrosstalkSources_1031141_31.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/0ea47cff-c54f-4111-b578-5e45eff8d333_isrForCrosstalkSources_1031141_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:01 proc_lsst.multi:146 [INFO] found job 17142645 in provider astro proc_lsst.multi INFO: found job 17142645 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd02bb22fd1f03f604e5f18886d1733 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/4cc63008-cbd6-4e1b-b76e-f019787f44d8_isrForCrosstalkSources_1031140_37.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/4cc63008-cbd6-4e1b-b76e-f019787f44d8_isrForCrosstalkSources_1031140_37.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a96b74f913aea1b253eceecec35afb15 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/453bc840-eb65-4245-949a-7284c0cc6d6a_isrForCrosstalkSources_1031141_61.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/453bc840-eb65-4245-949a-7284c0cc6d6a_isrForCrosstalkSources_1031141_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:01 proc_lsst.multi:146 [INFO] found job 17142646 in provider astro proc_lsst.multi INFO: found job 17142646 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:01 proc_lsst.multi:146 [INFO] found job 17142647 in provider astro proc_lsst.multi INFO: found job 17142647 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:01 proc_lsst.multi:146 [INFO] found job 17142648 in provider astro proc_lsst.multi INFO: found job 17142648 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:01 proc_lsst.multi:146 [INFO] found job 17142650 in provider astro proc_lsst.multi INFO: found job 17142650 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142651 in provider astro proc_lsst.multi INFO: found job 17142651 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e2c2207773dbfb7cf103e269d051479 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eec9986-4587-4d3f-bd6a-228395d7d3b7_isrForCrosstalkSources_1031140_55.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/8eec9986-4587-4d3f-bd6a-228395d7d3b7_isrForCrosstalkSources_1031140_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142652 in provider astro proc_lsst.multi INFO: found job 17142652 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142653 in provider astro proc_lsst.multi INFO: found job 17142653 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142654 in provider astro proc_lsst.multi INFO: found job 17142654 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142655 in provider astro proc_lsst.multi INFO: found job 17142655 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:02 proc_lsst.multi:146 [INFO] found job 17142656 in provider astro proc_lsst.multi INFO: found job 17142656 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:03 proc_lsst.multi:146 [INFO] found job 17142657 in provider astro proc_lsst.multi INFO: found job 17142657 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:03 proc_lsst.multi:146 [INFO] found job 17142658 in provider astro proc_lsst.multi INFO: found job 17142658 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5453bd57481c3b9092c716a08b6b1545 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49a520ae-744d-4179-854b-3a5f363a11f6_isrForCrosstalkSources_1031139_59.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/49a520ae-744d-4179-854b-3a5f363a11f6_isrForCrosstalkSources_1031139_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:03 proc_lsst.multi:146 [INFO] found job 17142659 in provider astro proc_lsst.multi INFO: found job 17142659 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:03 proc_lsst.multi:146 [INFO] found job 17142660 in provider astro proc_lsst.multi INFO: found job 17142660 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:03 proc_lsst.multi:146 [INFO] found job 17142661 in provider astro proc_lsst.multi INFO: found job 17142661 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:04 proc_lsst.multi:146 [INFO] found job 17142662 in provider astro proc_lsst.multi INFO: found job 17142662 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:04 proc_lsst.multi:146 [INFO] found job 17142663 in provider astro proc_lsst.multi INFO: found job 17142663 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01a5b3697f9a1cdd00fc4b0501262dac with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c7a7b2ec-9172-49e6-a690-e37ad4d13889_isrForCrosstalkSources_1031141_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/c7a7b2ec-9172-49e6-a690-e37ad4d13889_isrForCrosstalkSources_1031141_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:04 proc_lsst.multi:146 [INFO] found job 17142664 in provider astro proc_lsst.multi INFO: found job 17142664 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9cfeff95580c784a606888b12a18394 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e3fb814a-c1f0-4804-9e18-e854ebf8cb92_isrForCrosstalkSources_1031140_22.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/e3fb814a-c1f0-4804-9e18-e854ebf8cb92_isrForCrosstalkSources_1031140_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:04 proc_lsst.multi:146 [INFO] found job 17142665 in provider astro proc_lsst.multi INFO: found job 17142665 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d12a182b16ac573b1caf36f7eed16b0 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d546f6a7-a708-4c23-8c25-b2f279a9aea1_isrForCrosstalkSources_1031141_54.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/d546f6a7-a708-4c23-8c25-b2f279a9aea1_isrForCrosstalkSources_1031141_54.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819f6110f59f35200b1fbe1371b3a886 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f96c9810-b81e-42ba-83a2-44751020a7ed_isrForCrosstalkSources_1031140_47.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031140/f96c9810-b81e-42ba-83a2-44751020a7ed_isrForCrosstalkSources_1031140_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:04 proc_lsst.multi:146 [INFO] found job 17142666 in provider astro proc_lsst.multi INFO: found job 17142666 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fcc8fe3fd99babfa04f66b47c690b71 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/61f9e92c-f27f-4660-bb75-ca5bbeac4ede_isrForCrosstalkSources_1031139_62.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031139/61f9e92c-f27f-4660-bb75-ca5bbeac4ede_isrForCrosstalkSources_1031139_62.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c52735dc4efa8a5ddb16c21fbb47bc23 with result from task 181 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 186 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/675d1bc9-9573-4aa5-aa3e-1b2e8e9174a8_isrForCrosstalkSources_1031141_59.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/submit/DEEP/20210908/B1c/science#step0/20240317T012544Z/logs/isrForCrosstalkSources/1031141/675d1bc9-9573-4aa5-aa3e-1b2e8e9174a8_isrForCrosstalkSources_1031141_59.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:05 proc_lsst.multi:146 [INFO] found job 17142667 in provider astro proc_lsst.multi INFO: found job 17142667 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:05 proc_lsst.multi:146 [INFO] found job 17142668 in provider astro proc_lsst.multi INFO: found job 17142668 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142638,17142639,17142640,17142641,17142642,17142643,17142644,17142645,17142646,17142647,17142648,17142650,17142651,17142652,17142653,17142654,17142655,17142656,17142657,17142658,17142659,17142660,17142661,17142662,17142663,17142664,17142665,17142666,17142667,17142668' parsl.providers.slurm.slurm DEBUG: squeue returned 17142668 R 17142664 R 17142665 R 17142666 R 17142667 R 17142662 R 17142663 R 17142652 R 17142653 R 17142654 R 17142655 R 17142656 R 17142657 R 17142658 R 17142659 R 17142660 R 17142661 R 17142650 R 17142651 R 17142642 R 17142643 R 17142644 R 17142645 R 17142646 R 17142647 R 17142648 R 17142638 R 17142639 R 17142640 R 17142641 R parsl.providers.slurm.slurm DEBUG: Updating job 17142668 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142664 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142665 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142666 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142667 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142662 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142663 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142652 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142653 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142654 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142655 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142656 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142657 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142658 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142659 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142660 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142661 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142650 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142651 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142642 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142643 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142644 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142645 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142646 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142647 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142648 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142638 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142639 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142640 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142641 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:05 proc_lsst.multi:146 [INFO] found job 17142669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142669 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:05 proc_lsst.multi:146 [INFO] found job 17142670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142670 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:05 proc_lsst.multi:146 [INFO] found job 17142671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142671 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:06 proc_lsst.multi:146 [INFO] found job 17142672 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142672 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:06 proc_lsst.multi:146 [INFO] found job 17142673 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142673 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142669,17142670,17142671,17142672,17142673' parsl.providers.slurm.slurm DEBUG: squeue returned 17142669 R 17142670 R 17142671 R 17142672 R 17142673 R parsl.providers.slurm.slurm DEBUG: Updating job 17142669 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142670 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142671 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142672 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142673 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:06 proc_lsst.multi:146 [INFO] found job 17142674 in provider escience_a40 proc_lsst.multi INFO: found job 17142674 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:06 proc_lsst.multi:146 [INFO] found job 17142675 in provider escience_a40 proc_lsst.multi INFO: found job 17142675 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:06 proc_lsst.multi:146 [INFO] found job 17142676 in provider escience_a40 proc_lsst.multi INFO: found job 17142676 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:07 proc_lsst.multi:146 [INFO] found job 17142677 in provider escience_a40 proc_lsst.multi INFO: found job 17142677 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:07 proc_lsst.multi:146 [INFO] found job 17142678 in provider escience_a40 proc_lsst.multi INFO: found job 17142678 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:07 proc_lsst.multi:146 [INFO] found job 17142683 in provider escience_a40 proc_lsst.multi INFO: found job 17142683 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:07 proc_lsst.multi:146 [INFO] found job 17142687 in provider escience_a40 proc_lsst.multi INFO: found job 17142687 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:07 proc_lsst.multi:146 [INFO] found job 17142688 in provider escience_a40 proc_lsst.multi INFO: found job 17142688 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:08 proc_lsst.multi:146 [INFO] found job 17142689 in provider escience_a40 proc_lsst.multi INFO: found job 17142689 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING 2024-03-17 01:28:08 proc_lsst.multi:146 [INFO] found job 17142690 in provider escience_a40 proc_lsst.multi INFO: found job 17142690 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17142674,17142675,17142676,17142677,17142678,17142683,17142687,17142688,17142689,17142690' parsl.providers.slurm.slurm DEBUG: squeue returned 17142690 PD 17142689 R 17142688 R 17142674 R 17142675 R 17142676 R 17142677 R 17142678 R 17142683 R 17142687 R parsl.providers.slurm.slurm DEBUG: Updating job 17142690 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 17142689 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142688 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142674 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142675 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142676 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142677 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142678 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142683 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17142687 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 46, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 45/1 running/pending blocks, and 46 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (46) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 2.1457672119140625e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23096771977296 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=46, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 45 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7e8afb5060c4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7e8afb5060c4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ccfab2e226be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ccfab2e226be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b33260303bb6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b33260303bb6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3e821bcbe79 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3e821bcbe79 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c454900d48bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c454900d48bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f01edd390cb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f01edd390cb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5f080449542e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5f080449542e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7a5bdb61bb90 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7a5bdb61bb90 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1fc1e0523208 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1fc1e0523208 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a13e9cd92069 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a13e9cd92069 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7d6d50c55f0e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7d6d50c55f0e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2780877e6ecb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2780877e6ecb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7853a7a562f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7853a7a562f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 33a53ff36d0a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 33a53ff36d0a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5048de5d7ea5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5048de5d7ea5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0e33388c410d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0e33388c410d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a5d94bd26313 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a5d94bd26313 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1f5c13014927 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1f5c13014927 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 29c042eb8691 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 29c042eb8691 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea68667745c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea68667745c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cea9e0f0250f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cea9e0f0250f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f48e8fc9e074 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f48e8fc9e074 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e81cb917286 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e81cb917286 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ed869e784d09 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ed869e784d09 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a4503bd5ae7f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a4503bd5ae7f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8dd1a21a32c9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8dd1a21a32c9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3c37b77e46be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3c37b77e46be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 863027573a6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 863027573a6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19b788334ee4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19b788334ee4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9422cb570f65 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9422cb570f65 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a8856dd164ed parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a8856dd164ed parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01cf8bf7a3e3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01cf8bf7a3e3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bd504fc08aa7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bd504fc08aa7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 12a128f86ea1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 12a128f86ea1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 135432b14fec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 135432b14fec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3fcdd3631412 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3fcdd3631412 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 34de1695bd12 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 34de1695bd12 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a6bf94b2e523 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a6bf94b2e523 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 03fc044a4613 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 03fc044a4613 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9a23ca20bc2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9a23ca20bc2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e54e5cb01c25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e54e5cb01c25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5e28cdae8b3e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5e28cdae8b3e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8631c777e252 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8631c777e252 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 046a0160e04c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 046a0160e04c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 228374c3825c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 228374c3825c 2024-03-17 01:28:08 proc_lsst.multi:146 [INFO] found job 17142645 in provider astro proc_lsst.multi INFO: found job 17142645 in provider astro 2024-03-17 01:28:08 proc_lsst.multi:201 [INFO] cancelling 17142645 on provider astro proc_lsst.multi INFO: cancelling 17142645 on provider astro 2024-03-17 01:28:08 proc_lsst.multi:146 [INFO] found job 17142646 in provider astro proc_lsst.multi INFO: found job 17142646 in provider astro 2024-03-17 01:28:08 proc_lsst.multi:201 [INFO] cancelling 17142646 on provider astro proc_lsst.multi INFO: cancelling 17142646 on provider astro 2024-03-17 01:28:08 proc_lsst.multi:146 [INFO] found job 17142638 in provider astro proc_lsst.multi INFO: found job 17142638 in provider astro 2024-03-17 01:28:08 proc_lsst.multi:201 [INFO] cancelling 17142638 on provider astro proc_lsst.multi INFO: cancelling 17142638 on provider astro 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142650 in provider astro proc_lsst.multi INFO: found job 17142650 in provider astro 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142650 on provider astro proc_lsst.multi INFO: cancelling 17142650 on provider astro 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142651 in provider astro proc_lsst.multi INFO: found job 17142651 in provider astro 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142651 on provider astro proc_lsst.multi INFO: cancelling 17142651 on provider astro 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142641 in provider astro proc_lsst.multi INFO: found job 17142641 in provider astro 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142641 on provider astro proc_lsst.multi INFO: cancelling 17142641 on provider astro 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142683 in provider escience_a40 proc_lsst.multi INFO: found job 17142683 in provider escience_a40 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142683 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142683 on provider escience_a40 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142687 in provider escience_a40 proc_lsst.multi INFO: found job 17142687 in provider escience_a40 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142687 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142687 on provider escience_a40 2024-03-17 01:28:09 proc_lsst.multi:146 [INFO] found job 17142658 in provider astro proc_lsst.multi INFO: found job 17142658 in provider astro 2024-03-17 01:28:09 proc_lsst.multi:201 [INFO] cancelling 17142658 on provider astro proc_lsst.multi INFO: cancelling 17142658 on provider astro 2024-03-17 01:28:10 proc_lsst.multi:146 [INFO] found job 17142677 in provider escience_a40 proc_lsst.multi INFO: found job 17142677 in provider escience_a40 2024-03-17 01:28:10 proc_lsst.multi:201 [INFO] cancelling 17142677 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142677 on provider escience_a40 2024-03-17 01:28:10 proc_lsst.multi:146 [INFO] found job 17142674 in provider escience_a40 proc_lsst.multi INFO: found job 17142674 in provider escience_a40 2024-03-17 01:28:10 proc_lsst.multi:201 [INFO] cancelling 17142674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142674 on provider escience_a40 2024-03-17 01:28:10 proc_lsst.multi:146 [INFO] found job 17142664 in provider astro proc_lsst.multi INFO: found job 17142664 in provider astro 2024-03-17 01:28:10 proc_lsst.multi:201 [INFO] cancelling 17142664 on provider astro proc_lsst.multi INFO: cancelling 17142664 on provider astro 2024-03-17 01:28:10 proc_lsst.multi:146 [INFO] found job 17142656 in provider astro proc_lsst.multi INFO: found job 17142656 in provider astro 2024-03-17 01:28:10 proc_lsst.multi:201 [INFO] cancelling 17142656 on provider astro proc_lsst.multi INFO: cancelling 17142656 on provider astro 2024-03-17 01:28:10 proc_lsst.multi:146 [INFO] found job 17142668 in provider astro proc_lsst.multi INFO: found job 17142668 in provider astro 2024-03-17 01:28:10 proc_lsst.multi:201 [INFO] cancelling 17142668 on provider astro proc_lsst.multi INFO: cancelling 17142668 on provider astro 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142652 in provider astro proc_lsst.multi INFO: found job 17142652 in provider astro 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142652 on provider astro proc_lsst.multi INFO: cancelling 17142652 on provider astro 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142653 in provider astro proc_lsst.multi INFO: found job 17142653 in provider astro 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142653 on provider astro proc_lsst.multi INFO: cancelling 17142653 on provider astro 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 53465 on provider local proc_lsst.multi INFO: cancelling 53465 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53465 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142669 in provider escience_rtx6k 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142669 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142669 on provider escience_rtx6k 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142662 in provider astro proc_lsst.multi INFO: found job 17142662 in provider astro 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142662 on provider astro proc_lsst.multi INFO: cancelling 17142662 on provider astro 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142666 in provider astro proc_lsst.multi INFO: found job 17142666 in provider astro 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142666 on provider astro proc_lsst.multi INFO: cancelling 17142666 on provider astro 2024-03-17 01:28:11 proc_lsst.multi:146 [INFO] found job 17142663 in provider astro proc_lsst.multi INFO: found job 17142663 in provider astro 2024-03-17 01:28:11 proc_lsst.multi:201 [INFO] cancelling 17142663 on provider astro proc_lsst.multi INFO: cancelling 17142663 on provider astro 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142660 in provider astro proc_lsst.multi INFO: found job 17142660 in provider astro 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142660 on provider astro proc_lsst.multi INFO: cancelling 17142660 on provider astro 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142655 in provider astro proc_lsst.multi INFO: found job 17142655 in provider astro 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142655 on provider astro proc_lsst.multi INFO: cancelling 17142655 on provider astro 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142675 in provider escience_a40 proc_lsst.multi INFO: found job 17142675 in provider escience_a40 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142675 on provider escience_a40 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142661 in provider astro proc_lsst.multi INFO: found job 17142661 in provider astro 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142661 on provider astro proc_lsst.multi INFO: cancelling 17142661 on provider astro 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142665 in provider astro proc_lsst.multi INFO: found job 17142665 in provider astro 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142665 on provider astro proc_lsst.multi INFO: cancelling 17142665 on provider astro 2024-03-17 01:28:12 proc_lsst.multi:146 [INFO] found job 17142659 in provider astro proc_lsst.multi INFO: found job 17142659 in provider astro 2024-03-17 01:28:12 proc_lsst.multi:201 [INFO] cancelling 17142659 on provider astro proc_lsst.multi INFO: cancelling 17142659 on provider astro 2024-03-17 01:28:13 proc_lsst.multi:146 [INFO] found job 17142654 in provider astro proc_lsst.multi INFO: found job 17142654 in provider astro 2024-03-17 01:28:13 proc_lsst.multi:201 [INFO] cancelling 17142654 on provider astro proc_lsst.multi INFO: cancelling 17142654 on provider astro 2024-03-17 01:28:13 proc_lsst.multi:146 [INFO] found job 17142667 in provider astro proc_lsst.multi INFO: found job 17142667 in provider astro 2024-03-17 01:28:13 proc_lsst.multi:201 [INFO] cancelling 17142667 on provider astro proc_lsst.multi INFO: cancelling 17142667 on provider astro 2024-03-17 01:28:13 proc_lsst.multi:146 [INFO] found job 17142676 in provider escience_a40 proc_lsst.multi INFO: found job 17142676 in provider escience_a40 2024-03-17 01:28:13 proc_lsst.multi:201 [INFO] cancelling 17142676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142676 on provider escience_a40 2024-03-17 01:28:13 proc_lsst.multi:146 [INFO] found job 17142678 in provider escience_a40 proc_lsst.multi INFO: found job 17142678 in provider escience_a40 2024-03-17 01:28:13 proc_lsst.multi:201 [INFO] cancelling 17142678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142678 on provider escience_a40 2024-03-17 01:28:13 proc_lsst.multi:146 [INFO] found job 17142657 in provider astro proc_lsst.multi INFO: found job 17142657 in provider astro 2024-03-17 01:28:13 proc_lsst.multi:201 [INFO] cancelling 17142657 on provider astro proc_lsst.multi INFO: cancelling 17142657 on provider astro 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142670 in provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142670 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142670 on provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142671 in provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142671 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142671 on provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142673 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142673 in provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142673 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142673 on provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142672 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142672 in provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142672 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142672 on provider escience_rtx6k 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142689 in provider escience_a40 proc_lsst.multi INFO: found job 17142689 in provider escience_a40 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142689 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142689 on provider escience_a40 2024-03-17 01:28:14 proc_lsst.multi:146 [INFO] found job 17142688 in provider escience_a40 proc_lsst.multi INFO: found job 17142688 in provider escience_a40 2024-03-17 01:28:14 proc_lsst.multi:201 [INFO] cancelling 17142688 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142688 on provider escience_a40 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142642 in provider astro proc_lsst.multi INFO: found job 17142642 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142642 on provider astro proc_lsst.multi INFO: cancelling 17142642 on provider astro 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142647 in provider astro proc_lsst.multi INFO: found job 17142647 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142647 on provider astro proc_lsst.multi INFO: cancelling 17142647 on provider astro 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142639 in provider astro proc_lsst.multi INFO: found job 17142639 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142639 on provider astro proc_lsst.multi INFO: cancelling 17142639 on provider astro 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142644 in provider astro proc_lsst.multi INFO: found job 17142644 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142644 on provider astro proc_lsst.multi INFO: cancelling 17142644 on provider astro 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142648 in provider astro proc_lsst.multi INFO: found job 17142648 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142648 on provider astro proc_lsst.multi INFO: cancelling 17142648 on provider astro 2024-03-17 01:28:15 proc_lsst.multi:146 [INFO] found job 17142640 in provider astro proc_lsst.multi INFO: found job 17142640 in provider astro 2024-03-17 01:28:15 proc_lsst.multi:201 [INFO] cancelling 17142640 on provider astro proc_lsst.multi INFO: cancelling 17142640 on provider astro 2024-03-17 01:28:16 proc_lsst.multi:146 [INFO] found job 17142643 in provider astro proc_lsst.multi INFO: found job 17142643 in provider astro 2024-03-17 01:28:16 proc_lsst.multi:201 [INFO] cancelling 17142643 on provider astro proc_lsst.multi INFO: cancelling 17142643 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-17 01:28:16 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-17 01:28:16 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17142638', '17142639', '17142640', '17142641', '17142642', '17142643', '17142644', '17142645', '17142646', '17142647', '17142648', '17142650', '17142651', '17142652', '17142653', '17142654', '17142655', '17142656', '17142657', '17142658', '17142659', '17142660', '17142661', '17142662', '17142663', '17142664', '17142665', '17142666', '17142667', '17142668', '17142674', '17142675', '17142676', '17142677', '17142678', '17142683', '17142687', '17142688', '17142689', '17142690', '17142669', '17142670', '17142671', '17142672', '17142673', '53465'] proc_lsst.multi INFO: new jobs since last cancel ['17142638', '17142639', '17142640', '17142641', '17142642', '17142643', '17142644', '17142645', '17142646', '17142647', '17142648', '17142650', '17142651', '17142652', '17142653', '17142654', '17142655', '17142656', '17142657', '17142658', '17142659', '17142660', '17142661', '17142662', '17142663', '17142664', '17142665', '17142666', '17142667', '17142668', '17142674', '17142675', '17142676', '17142677', '17142678', '17142683', '17142687', '17142688', '17142689', '17142690', '17142669', '17142670', '17142671', '17142672', '17142673', '53465'] 2024-03-17 01:28:16 proc_lsst.multi:146 [INFO] found job 17142638 in provider astro proc_lsst.multi INFO: found job 17142638 in provider astro 2024-03-17 01:28:16 proc_lsst.multi:201 [INFO] cancelling 17142638 on provider astro proc_lsst.multi INFO: cancelling 17142638 on provider astro 2024-03-17 01:28:16 proc_lsst.multi:146 [INFO] found job 17142639 in provider astro proc_lsst.multi INFO: found job 17142639 in provider astro 2024-03-17 01:28:16 proc_lsst.multi:201 [INFO] cancelling 17142639 on provider astro proc_lsst.multi INFO: cancelling 17142639 on provider astro 2024-03-17 01:28:16 proc_lsst.multi:146 [INFO] found job 17142640 in provider astro proc_lsst.multi INFO: found job 17142640 in provider astro 2024-03-17 01:28:16 proc_lsst.multi:201 [INFO] cancelling 17142640 on provider astro proc_lsst.multi INFO: cancelling 17142640 on provider astro 2024-03-17 01:28:16 proc_lsst.multi:146 [INFO] found job 17142641 in provider astro proc_lsst.multi INFO: found job 17142641 in provider astro 2024-03-17 01:28:16 proc_lsst.multi:201 [INFO] cancelling 17142641 on provider astro proc_lsst.multi INFO: cancelling 17142641 on provider astro 2024-03-17 01:28:17 proc_lsst.multi:146 [INFO] found job 17142642 in provider astro proc_lsst.multi INFO: found job 17142642 in provider astro 2024-03-17 01:28:17 proc_lsst.multi:201 [INFO] cancelling 17142642 on provider astro proc_lsst.multi INFO: cancelling 17142642 on provider astro 2024-03-17 01:28:17 proc_lsst.multi:146 [INFO] found job 17142643 in provider astro proc_lsst.multi INFO: found job 17142643 in provider astro 2024-03-17 01:28:17 proc_lsst.multi:201 [INFO] cancelling 17142643 on provider astro proc_lsst.multi INFO: cancelling 17142643 on provider astro 2024-03-17 01:28:17 proc_lsst.multi:146 [INFO] found job 17142644 in provider astro proc_lsst.multi INFO: found job 17142644 in provider astro 2024-03-17 01:28:17 proc_lsst.multi:201 [INFO] cancelling 17142644 on provider astro proc_lsst.multi INFO: cancelling 17142644 on provider astro 2024-03-17 01:28:17 proc_lsst.multi:146 [INFO] found job 17142645 in provider astro proc_lsst.multi INFO: found job 17142645 in provider astro 2024-03-17 01:28:17 proc_lsst.multi:201 [INFO] cancelling 17142645 on provider astro proc_lsst.multi INFO: cancelling 17142645 on provider astro 2024-03-17 01:28:17 proc_lsst.multi:146 [INFO] found job 17142646 in provider astro proc_lsst.multi INFO: found job 17142646 in provider astro 2024-03-17 01:28:17 proc_lsst.multi:201 [INFO] cancelling 17142646 on provider astro proc_lsst.multi INFO: cancelling 17142646 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142647 in provider astro proc_lsst.multi INFO: found job 17142647 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142647 on provider astro proc_lsst.multi INFO: cancelling 17142647 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142648 in provider astro proc_lsst.multi INFO: found job 17142648 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142648 on provider astro proc_lsst.multi INFO: cancelling 17142648 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142650 in provider astro proc_lsst.multi INFO: found job 17142650 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142650 on provider astro proc_lsst.multi INFO: cancelling 17142650 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142651 in provider astro proc_lsst.multi INFO: found job 17142651 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142651 on provider astro proc_lsst.multi INFO: cancelling 17142651 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142652 in provider astro proc_lsst.multi INFO: found job 17142652 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142652 on provider astro proc_lsst.multi INFO: cancelling 17142652 on provider astro 2024-03-17 01:28:18 proc_lsst.multi:146 [INFO] found job 17142653 in provider astro proc_lsst.multi INFO: found job 17142653 in provider astro 2024-03-17 01:28:18 proc_lsst.multi:201 [INFO] cancelling 17142653 on provider astro proc_lsst.multi INFO: cancelling 17142653 on provider astro 2024-03-17 01:28:19 proc_lsst.multi:146 [INFO] found job 17142654 in provider astro proc_lsst.multi INFO: found job 17142654 in provider astro 2024-03-17 01:28:19 proc_lsst.multi:201 [INFO] cancelling 17142654 on provider astro proc_lsst.multi INFO: cancelling 17142654 on provider astro 2024-03-17 01:28:19 proc_lsst.multi:146 [INFO] found job 17142655 in provider astro proc_lsst.multi INFO: found job 17142655 in provider astro 2024-03-17 01:28:19 proc_lsst.multi:201 [INFO] cancelling 17142655 on provider astro proc_lsst.multi INFO: cancelling 17142655 on provider astro 2024-03-17 01:28:19 proc_lsst.multi:146 [INFO] found job 17142656 in provider astro proc_lsst.multi INFO: found job 17142656 in provider astro 2024-03-17 01:28:19 proc_lsst.multi:201 [INFO] cancelling 17142656 on provider astro proc_lsst.multi INFO: cancelling 17142656 on provider astro 2024-03-17 01:28:19 proc_lsst.multi:146 [INFO] found job 17142657 in provider astro proc_lsst.multi INFO: found job 17142657 in provider astro 2024-03-17 01:28:19 proc_lsst.multi:201 [INFO] cancelling 17142657 on provider astro proc_lsst.multi INFO: cancelling 17142657 on provider astro 2024-03-17 01:28:19 proc_lsst.multi:146 [INFO] found job 17142658 in provider astro proc_lsst.multi INFO: found job 17142658 in provider astro 2024-03-17 01:28:19 proc_lsst.multi:201 [INFO] cancelling 17142658 on provider astro proc_lsst.multi INFO: cancelling 17142658 on provider astro 2024-03-17 01:28:20 proc_lsst.multi:146 [INFO] found job 17142659 in provider astro proc_lsst.multi INFO: found job 17142659 in provider astro 2024-03-17 01:28:20 proc_lsst.multi:201 [INFO] cancelling 17142659 on provider astro proc_lsst.multi INFO: cancelling 17142659 on provider astro 2024-03-17 01:28:20 proc_lsst.multi:146 [INFO] found job 17142660 in provider astro proc_lsst.multi INFO: found job 17142660 in provider astro 2024-03-17 01:28:20 proc_lsst.multi:201 [INFO] cancelling 17142660 on provider astro proc_lsst.multi INFO: cancelling 17142660 on provider astro 2024-03-17 01:28:20 proc_lsst.multi:146 [INFO] found job 17142661 in provider astro proc_lsst.multi INFO: found job 17142661 in provider astro 2024-03-17 01:28:20 proc_lsst.multi:201 [INFO] cancelling 17142661 on provider astro proc_lsst.multi INFO: cancelling 17142661 on provider astro 2024-03-17 01:28:20 proc_lsst.multi:146 [INFO] found job 17142662 in provider astro proc_lsst.multi INFO: found job 17142662 in provider astro 2024-03-17 01:28:20 proc_lsst.multi:201 [INFO] cancelling 17142662 on provider astro proc_lsst.multi INFO: cancelling 17142662 on provider astro 2024-03-17 01:28:20 proc_lsst.multi:146 [INFO] found job 17142663 in provider astro proc_lsst.multi INFO: found job 17142663 in provider astro 2024-03-17 01:28:20 proc_lsst.multi:201 [INFO] cancelling 17142663 on provider astro proc_lsst.multi INFO: cancelling 17142663 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142664 in provider astro proc_lsst.multi INFO: found job 17142664 in provider astro 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142664 on provider astro proc_lsst.multi INFO: cancelling 17142664 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142665 in provider astro proc_lsst.multi INFO: found job 17142665 in provider astro 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142665 on provider astro proc_lsst.multi INFO: cancelling 17142665 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142666 in provider astro proc_lsst.multi INFO: found job 17142666 in provider astro 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142666 on provider astro proc_lsst.multi INFO: cancelling 17142666 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142667 in provider astro proc_lsst.multi INFO: found job 17142667 in provider astro 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142667 on provider astro proc_lsst.multi INFO: cancelling 17142667 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142668 in provider astro proc_lsst.multi INFO: found job 17142668 in provider astro 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142668 on provider astro proc_lsst.multi INFO: cancelling 17142668 on provider astro 2024-03-17 01:28:21 proc_lsst.multi:146 [INFO] found job 17142674 in provider escience_a40 proc_lsst.multi INFO: found job 17142674 in provider escience_a40 2024-03-17 01:28:21 proc_lsst.multi:201 [INFO] cancelling 17142674 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142674 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142675 in provider escience_a40 proc_lsst.multi INFO: found job 17142675 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142675 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142675 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142676 in provider escience_a40 proc_lsst.multi INFO: found job 17142676 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142676 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142676 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142677 in provider escience_a40 proc_lsst.multi INFO: found job 17142677 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142677 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142677 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142678 in provider escience_a40 proc_lsst.multi INFO: found job 17142678 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142678 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142678 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142683 in provider escience_a40 proc_lsst.multi INFO: found job 17142683 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142683 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142683 on provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:146 [INFO] found job 17142687 in provider escience_a40 proc_lsst.multi INFO: found job 17142687 in provider escience_a40 2024-03-17 01:28:22 proc_lsst.multi:201 [INFO] cancelling 17142687 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142687 on provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142688 in provider escience_a40 proc_lsst.multi INFO: found job 17142688 in provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142688 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142688 on provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142689 in provider escience_a40 proc_lsst.multi INFO: found job 17142689 in provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142689 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142689 on provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142690 in provider escience_a40 proc_lsst.multi INFO: found job 17142690 in provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142690 on provider escience_a40 proc_lsst.multi INFO: cancelling 17142690 on provider escience_a40 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142669 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142669 in provider escience_rtx6k 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142669 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142669 on provider escience_rtx6k 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142670 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142670 in provider escience_rtx6k 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142670 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142670 on provider escience_rtx6k 2024-03-17 01:28:23 proc_lsst.multi:146 [INFO] found job 17142671 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142671 in provider escience_rtx6k 2024-03-17 01:28:23 proc_lsst.multi:201 [INFO] cancelling 17142671 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142671 on provider escience_rtx6k 2024-03-17 01:28:24 proc_lsst.multi:146 [INFO] found job 17142672 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142672 in provider escience_rtx6k 2024-03-17 01:28:24 proc_lsst.multi:201 [INFO] cancelling 17142672 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142672 on provider escience_rtx6k 2024-03-17 01:28:24 proc_lsst.multi:146 [INFO] found job 17142673 in provider escience_rtx6k proc_lsst.multi INFO: found job 17142673 in provider escience_rtx6k 2024-03-17 01:28:24 proc_lsst.multi:201 [INFO] cancelling 17142673 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17142673 on provider escience_rtx6k 2024-03-17 01:28:24 proc_lsst.multi:146 [INFO] found job 53465 in provider local proc_lsst.multi INFO: found job 53465 in provider local 2024-03-17 01:28:24 proc_lsst.multi:201 [INFO] cancelling 53465 on provider local proc_lsst.multi INFO: cancelling 53465 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 53465 parsl.providers.local.local WARNING: Failed to kill PID: 53465 and child processes on local 2024-03-17 01:28:25 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 137.4196 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20210908_B1c_science#step0_20240317T012544Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 137.4256 seconds; current memory usage: 0.276 Gibyte, delta: 0.002 Gibyte, peak delta: 0.004 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 156.0324 seconds; current memory usage: 0.276 Gibyte, delta: 0.137 Gibyte, peak delta: 0.139 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.278 Gibyte (main), 0.278 Gibyte (largest child process) Run Id: None Run Name: DEEP_20210908_B1c_science#step0_20240317T012544Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m37.930s user 0m33.034s sys 0m15.347s