+ PROC_LSST_SUBMIT_PATH=/mmfs1/home/stevengs/dirac/DEEP/submit + bps submit /gscratch/dirac/shared/opt/proc_lsst/pipelines/submit.yaml -b /mmfs1/home/stevengs/dirac/DEEP/repo -i DEEP/20201019/B1a --output-run DEEP/20201019/B1a/science#step0/20240314T043249Z --qgraph pipeline.qgraph lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 2.6032 seconds; current memory usage: 0.180 Gibyte, delta: 0.043 Gibyte, peak delta: 0.055 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0064 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0137 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/pipeline_orig.qgraph DEEP/20201019/B1a/science#step0/20240314T043249Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 2.6397 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 7.0004 seconds; current memory usage: 0.277 Gibyte, delta: 0.097 Gibyte, peak delta: 0.085 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.267 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0133 seconds; current memory usage: 0.277 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.267 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 434 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20201019_B1a_science#step0_20240314T043249Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.0895 seconds; current memory usage: 0.277 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.277 Gibyte (main), 0.267 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 436 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.2731 seconds; current memory usage: 0.278 Gibyte, delta: 0.001 Gibyte, peak delta: 0.001 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.278 Gibyte (main), 0.267 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: b4bc0a4f-5551-425b-882d-ecabca5bb6cc parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-14 04:33:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-14 04:33:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 04:33:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-14 04:33:12 proc_lsst.multi:170 [INFO] [multi] job_id 98211 proc_lsst.multi INFO: [multi] job_id 98211 2024-03-14 04:33:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:12 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 98211 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 98211 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 98211 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash 10583cd2a99a66f4436fea01de469180 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e9290b37-5fa5-4bd0-9fc4-593e257dd945_isrForCrosstalkSources_946958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e9290b37-5fa5-4bd0-9fc4-593e257dd945_isrForCrosstalkSources_946958_19.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 257e8ce23a2d87c0f7c479703086baa1 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/12f56b25-106c-4896-ad49-4861e7d17352_isrForCrosstalkSources_946959_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/12f56b25-106c-4896-ad49-4861e7d17352_isrForCrosstalkSources_946959_58.stderr parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash ceef63fd09c9cea08a6c16c1dcb4e14d parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b22f1848-c8e5-4e62-984d-221db95874d3_isrForCrosstalkSources_946959_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b22f1848-c8e5-4e62-984d-221db95874d3_isrForCrosstalkSources_946959_34.stderr parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 96ed087a1ad85b8eab0a3e2a17279470 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/ab0b7c5f-1879-44b2-980c-84d25ff20a5c_isrForCrosstalkSources_946958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/ab0b7c5f-1879-44b2-980c-84d25ff20a5c_isrForCrosstalkSources_946958_43.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 0a676f03b35aacf822bfb34abff806e8 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/09f1a45f-58fe-4673-bb62-72ff1c757c81_isrForCrosstalkSources_946851_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/09f1a45f-58fe-4673-bb62-72ff1c757c81_isrForCrosstalkSources_946851_30.stderr parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash 1432ccb3352a7350fc92b657790b993d parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/81bf828e-f0a7-4b1d-921b-ce1db454d716_isrForCrosstalkSources_946957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/81bf828e-f0a7-4b1d-921b-ce1db454d716_isrForCrosstalkSources_946957_24.stderr parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash b329cab496213fed71d7a2749b50eae2 parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a59fed00-5920-4c4e-8ecf-89e3159b2587_isrForCrosstalkSources_946956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a59fed00-5920-4c4e-8ecf-89e3159b2587_isrForCrosstalkSources_946956_15.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 58acf0dfe92472350a4bedbc037fabc9 parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/85f122c3-68c5-40e2-8e06-67488aefd0f5_isrForCrosstalkSources_946851_44.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/85f122c3-68c5-40e2-8e06-67488aefd0f5_isrForCrosstalkSources_946851_44.stderr parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 8 has memoization hash a85ddc7aa8771c885af305ca733b1ad7 parsl.dataflow.memoization INFO: Task 8 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 8 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/dd020585-88b6-4a3c-88c8-a2d1d313de1c_isrForCrosstalkSources_946851_46.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/dd020585-88b6-4a3c-88c8-a2d1d313de1c_isrForCrosstalkSources_946851_46.stderr parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash 3a3bef5ddcbfafaafbf157942ac77caf parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0d104b3e-6b3f-42e9-9991-b2d41ee888f3_isrForCrosstalkSources_946852_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0d104b3e-6b3f-42e9-9991-b2d41ee888f3_isrForCrosstalkSources_946852_7.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash d3b1866a5ca9c25587c23d2d759385f4 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/45bb6713-380e-4fcd-934f-1ec03e05b9c5_isrForCrosstalkSources_946957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/45bb6713-380e-4fcd-934f-1ec03e05b9c5_isrForCrosstalkSources_946957_13.stderr parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 86da97bca341a3dde7a432f6e0e3a120 parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7171f884-e3f9-477c-8c12-d230470d1447_isrForCrosstalkSources_946853_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7171f884-e3f9-477c-8c12-d230470d1447_isrForCrosstalkSources_946853_30.stderr parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash 7f949d839a9d154cbc7df432032d3dc3 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b38f83b2-7431-4afe-a52b-4e0a328a9833_isrForCrosstalkSources_946957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b38f83b2-7431-4afe-a52b-4e0a328a9833_isrForCrosstalkSources_946957_12.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash c2885cda43120e5393b226109ba01200 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dd7dbf63-246a-4067-ada2-abd447f2e4e2_isrForCrosstalkSources_946959_17.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dd7dbf63-246a-4067-ada2-abd447f2e4e2_isrForCrosstalkSources_946959_17.stderr parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash e5b32ae97d297797de3df75ad645c48f parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/8543ec21-af6d-4963-8532-c4c464895522_isrForCrosstalkSources_946958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/8543ec21-af6d-4963-8532-c4c464895522_isrForCrosstalkSources_946958_16.stderr parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 460999aa42140cde789ac31b88622b3e parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/77072f22-6e49-49af-bcc9-8f5c680d72c6_isrForCrosstalkSources_946853_58.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/77072f22-6e49-49af-bcc9-8f5c680d72c6_isrForCrosstalkSources_946853_58.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 5f80a46010b68d36c9a7c50b0569360d parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/68b21b9e-fb51-4b53-bff6-dec4800f2daa_isrForCrosstalkSources_946957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/68b21b9e-fb51-4b53-bff6-dec4800f2daa_isrForCrosstalkSources_946957_54.stderr parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 17 has memoization hash 179993a1de436dd2f0cbb43ebe815f7f parsl.dataflow.memoization INFO: Task 17 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 17 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a50dc54c-dcce-4af3-a867-04d153de30a1_isrForCrosstalkSources_946958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a50dc54c-dcce-4af3-a867-04d153de30a1_isrForCrosstalkSources_946958_30.stderr parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash c815947bcdfbd9a9c18a7e72a897915e parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8a2eb305-086a-4cdd-9d0f-8ada85ec5963_isrForCrosstalkSources_946853_18.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8a2eb305-086a-4cdd-9d0f-8ada85ec5963_isrForCrosstalkSources_946853_18.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 321219af2439ba3dbefae6a96816288b parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/521166d5-e7c6-4c39-9548-3d585ef5ce9c_isrForCrosstalkSources_946852_3.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/521166d5-e7c6-4c39-9548-3d585ef5ce9c_isrForCrosstalkSources_946852_3.stderr parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash b1c484019c6a4dd27dd12f0b3c654e47 parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/c81d2413-5714-482c-a24d-56f66ffdc6e8_isrForCrosstalkSources_946851_11.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/c81d2413-5714-482c-a24d-56f66ffdc6e8_isrForCrosstalkSources_946851_11.stderr parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash c790ce79f4e56cf845a73750d5c2427e parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0483cb62-d1f6-46eb-a078-7f4f68204922_isrForCrosstalkSources_946852_44.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0483cb62-d1f6-46eb-a078-7f4f68204922_isrForCrosstalkSources_946852_44.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 44fc15cad83bbcdf9086b834ac90adad parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/80a3ec26-1ddb-46f4-9e5e-5afc7c90c0c9_isrForCrosstalkSources_946958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/80a3ec26-1ddb-46f4-9e5e-5afc7c90c0c9_isrForCrosstalkSources_946958_61.stderr parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash fccfa3f671d0752ed318de43459398df parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6092ed1-9e73-4c4c-a02d-cda54dac4def_isrForCrosstalkSources_946853_26.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6092ed1-9e73-4c4c-a02d-cda54dac4def_isrForCrosstalkSources_946853_26.stderr parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash ca7acb213585138f7ae90fde40315bef parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e1130ede-29ae-42fd-9191-dd64e8f2b86d_isrForCrosstalkSources_946957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e1130ede-29ae-42fd-9191-dd64e8f2b86d_isrForCrosstalkSources_946957_16.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash f27743dd85708ab93aeff9d4f8144016 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d5fe790a-ba26-422e-a137-75c03b068eef_isrForCrosstalkSources_946851_50.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d5fe790a-ba26-422e-a137-75c03b068eef_isrForCrosstalkSources_946851_50.stderr parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash d6dc1ce4d0d42adc87df73b6720c959e parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e13df8af-efbb-4779-9b7d-d932680f8c96_isrForCrosstalkSources_946959_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e13df8af-efbb-4779-9b7d-d932680f8c96_isrForCrosstalkSources_946959_9.stderr parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 8dca793bce165e233f986967325e1a8d parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d13a24ac-4e2c-4dcf-a8c1-f18be7e671f3_isrForCrosstalkSources_946956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d13a24ac-4e2c-4dcf-a8c1-f18be7e671f3_isrForCrosstalkSources_946956_48.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash b190fcf92d7a88f340927fb1b2ba5a32 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/23fb6e7d-31fa-4437-b1fe-6f0552b67ee0_isrForCrosstalkSources_946957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/23fb6e7d-31fa-4437-b1fe-6f0552b67ee0_isrForCrosstalkSources_946957_53.stderr parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 1be1e7f248f7c97dc498bea33fa39dd4 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7d37414c-cfbe-4c75-a40d-d951cac8ecdb_isrForCrosstalkSources_946852_61.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7d37414c-cfbe-4c75-a40d-d951cac8ecdb_isrForCrosstalkSources_946852_61.stderr parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash 2cbdf99bb80b7facf3d263f8b4214fc8 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/3a2b93fc-aae5-462e-8f33-2d8185d38724_isrForCrosstalkSources_946956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/3a2b93fc-aae5-462e-8f33-2d8185d38724_isrForCrosstalkSources_946956_52.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 993a80726dfcfc589aaf8217ac43cf46 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2d62fd8d-d6c7-4078-8d27-fb858fda5cdf_isrForCrosstalkSources_946957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2d62fd8d-d6c7-4078-8d27-fb858fda5cdf_isrForCrosstalkSources_946957_6.stderr parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash af0b70ad783a7b926814bbbd98fe24b5 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fc243564-f4eb-4ed8-941b-c28e4bf941b1_isrForCrosstalkSources_946956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fc243564-f4eb-4ed8-941b-c28e4bf941b1_isrForCrosstalkSources_946956_45.stderr parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash d1f013a48738ba524895e1f5db20aa55 parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8d9335e9-eb0d-44df-bbfa-ce1f65df1195_isrForCrosstalkSources_946959_45.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8d9335e9-eb0d-44df-bbfa-ce1f65df1195_isrForCrosstalkSources_946959_45.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash 1777f12030d103d897a6b6fefd19f8e3 parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a57fc90-ceaa-46a0-8290-e8753abb5390_isrForCrosstalkSources_946853_33.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a57fc90-ceaa-46a0-8290-e8753abb5390_isrForCrosstalkSources_946853_33.stderr parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 0821f4eefcaf2ca790e1bcdf3c6fa279 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/29210118-553f-46ec-a9a5-46757fba314c_isrForCrosstalkSources_946852_23.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/29210118-553f-46ec-a9a5-46757fba314c_isrForCrosstalkSources_946852_23.stderr parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash 41ebd1cb899579d58e5b78df6410e1a5 parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d17f1a29-074a-4285-9f3f-0aef881aec5b_isrForCrosstalkSources_946851_52.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d17f1a29-074a-4285-9f3f-0aef881aec5b_isrForCrosstalkSources_946851_52.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash 97a6723fe4d187e65f142b4457f6dd46 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/24eee0b2-6e20-4b5c-8def-738f26636dcd_isrForCrosstalkSources_946851_3.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/24eee0b2-6e20-4b5c-8def-738f26636dcd_isrForCrosstalkSources_946851_3.stderr parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 8cf41e08a4c26db6056509b96eb11e1b parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7ef3f91b-a5f3-45da-8a54-52f8b73281cf_isrForCrosstalkSources_946852_45.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7ef3f91b-a5f3-45da-8a54-52f8b73281cf_isrForCrosstalkSources_946852_45.stderr parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 29003628b1e6ae471e6491adb3b59213 parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a8b091c6-b231-4570-b43f-4cae46dec519_isrForCrosstalkSources_946956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a8b091c6-b231-4570-b43f-4cae46dec519_isrForCrosstalkSources_946956_61.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash a4e61221de9ed19d1add6454368334ba parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/ebad2e87-fcf7-4619-b608-67fd2f243fc5_isrForCrosstalkSources_946959_37.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/ebad2e87-fcf7-4619-b608-67fd2f243fc5_isrForCrosstalkSources_946959_37.stderr parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash 204f5c01b1c710c118af18157a705527 parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/635de5af-169e-41df-8275-197253a3c1b9_isrForCrosstalkSources_946958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/635de5af-169e-41df-8275-197253a3c1b9_isrForCrosstalkSources_946958_20.stderr parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash adefd47ba98739e2c96b53ec9f530819 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f712c01a-69f5-4f4f-bc51-70d7307c7614_isrForCrosstalkSources_946853_11.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f712c01a-69f5-4f4f-bc51-70d7307c7614_isrForCrosstalkSources_946853_11.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash 7df0945c16f7af4349f864bd69b90e1f parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b55da2fa-dc86-42da-9961-94c443c501da_isrForCrosstalkSources_946956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b55da2fa-dc86-42da-9961-94c443c501da_isrForCrosstalkSources_946956_6.stderr parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash 80739ce6a80809f28825479a7c21a52a parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/64dd05f0-0cec-4272-b781-4dbf9fc1b531_isrForCrosstalkSources_946956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/64dd05f0-0cec-4272-b781-4dbf9fc1b531_isrForCrosstalkSources_946956_41.stderr parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash d7e2b6f4cfaeb0b9fb96440e5950dafb parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/994c2669-a63e-498f-90f4-5701d88c6c3e_isrForCrosstalkSources_946958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/994c2669-a63e-498f-90f4-5701d88c6c3e_isrForCrosstalkSources_946958_41.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash c4e7de03c017ebf62d6e2838276cfc92 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b63a9bee-c3ec-410c-b797-27a76ba5584f_isrForCrosstalkSources_946956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b63a9bee-c3ec-410c-b797-27a76ba5584f_isrForCrosstalkSources_946956_21.stderr parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash 2f036e1a1056993a76954c1e2ca2a6c2 parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/285d4677-9966-49a3-9eb3-47a2e1fd03bf_isrForCrosstalkSources_946852_38.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/285d4677-9966-49a3-9eb3-47a2e1fd03bf_isrForCrosstalkSources_946852_38.stderr parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash c5faeda740fb32cb1288e7a0b21c2ec7 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7e522f20-2ae9-40b8-95e9-91b7d4a1730d_isrForCrosstalkSources_946958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7e522f20-2ae9-40b8-95e9-91b7d4a1730d_isrForCrosstalkSources_946958_2.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash ea723e406046fd3e3fcfbe9c0f4cdf4d parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cdd65df2-d425-4f5f-a375-760bc997cdc3_isrForCrosstalkSources_946853_54.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cdd65df2-d425-4f5f-a375-760bc997cdc3_isrForCrosstalkSources_946853_54.stderr parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 18ea3b5e200a7f8b294f388759b76053 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e07f42e-c418-4d0c-b968-54e3f25ff088_isrForCrosstalkSources_946959_50.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e07f42e-c418-4d0c-b968-54e3f25ff088_isrForCrosstalkSources_946959_50.stderr parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash a92f81f490ae5ce0ad0a3cf4ade07db9 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3fccc2a1-c7fd-4e79-a588-b442ba0efd19_isrForCrosstalkSources_946959_56.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3fccc2a1-c7fd-4e79-a588-b442ba0efd19_isrForCrosstalkSources_946959_56.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash f4e0d95770e67b015f21d5137747d2a5 parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6a839b9-90db-49a8-ada9-a93b0ae16374_isrForCrosstalkSources_946853_35.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6a839b9-90db-49a8-ada9-a93b0ae16374_isrForCrosstalkSources_946853_35.stderr parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash 532eeaf32f5ee6d3347c45c63e3582df parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a187315b-6263-4949-9120-8d2fa6927687_isrForCrosstalkSources_946959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a187315b-6263-4949-9120-8d2fa6927687_isrForCrosstalkSources_946959_2.stderr parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 26f9bbab548eddad71c7bdde3557f5a1 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eda54860-e98b-4d4a-9aca-8992e458866e_isrForCrosstalkSources_946851_38.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eda54860-e98b-4d4a-9aca-8992e458866e_isrForCrosstalkSources_946851_38.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 55158fce0b39bcb883af77a1edee026c parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a965af7b-b2a1-4fc1-871b-40d5fe13dac9_isrForCrosstalkSources_946853_20.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a965af7b-b2a1-4fc1-871b-40d5fe13dac9_isrForCrosstalkSources_946853_20.stderr parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash ce785687e0416474bec2e369037bd0d1 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/613e8338-3853-4140-a434-de0f217c7ee3_isrForCrosstalkSources_946958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/613e8338-3853-4140-a434-de0f217c7ee3_isrForCrosstalkSources_946958_6.stderr parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash d5f15e8bd742c2ee3d0eda74b0d161eb parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a3b2445-e54a-48a5-ae4b-f2aadf4c83ba_isrForCrosstalkSources_946957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a3b2445-e54a-48a5-ae4b-f2aadf4c83ba_isrForCrosstalkSources_946957_33.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 4096575ad18a1ff04d0161b62df53508 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8108ee8e-9f06-41d8-8bd0-e9b3febffa27_isrForCrosstalkSources_946853_3.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8108ee8e-9f06-41d8-8bd0-e9b3febffa27_isrForCrosstalkSources_946853_3.stderr parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash d476c04f8de7ddb457fbb89e3dea3087 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c6df45e4-9922-4610-979a-f3cefcae9056_isrForCrosstalkSources_946852_5.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c6df45e4-9922-4610-979a-f3cefcae9056_isrForCrosstalkSources_946852_5.stderr parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 97ebff8c4e88c10630c02a1287dc606f parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ff7bfff9-adb2-4820-9cd7-2ec5b30047c8_isrForCrosstalkSources_946956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ff7bfff9-adb2-4820-9cd7-2ec5b30047c8_isrForCrosstalkSources_946956_34.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 7e92609887c0e3c3a14b4e46fa1fad10 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cd796337-65bf-468e-8fe1-9aa730266465_isrForCrosstalkSources_946851_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cd796337-65bf-468e-8fe1-9aa730266465_isrForCrosstalkSources_946851_27.stderr parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 311170dab98dc928a511e9a7e70e9cf7 parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/85a6d340-611c-4aea-8d05-d130c06c899e_isrForCrosstalkSources_946853_37.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/85a6d340-611c-4aea-8d05-d130c06c899e_isrForCrosstalkSources_946853_37.stderr parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 2898246ee41af83127519b28e68ab3bb parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8c8a7d78-063e-4ade-ae86-168ac93dd1fd_isrForCrosstalkSources_946959_10.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8c8a7d78-063e-4ade-ae86-168ac93dd1fd_isrForCrosstalkSources_946959_10.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 61f482fc2ea2c1e8b252dec6e6613797 parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/98f66367-8e96-4bcc-9f76-a175e7356a91_isrForCrosstalkSources_946957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/98f66367-8e96-4bcc-9f76-a175e7356a91_isrForCrosstalkSources_946957_7.stderr parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash e62fdb2d6ef3fae01cbef669e8cce030 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f2ec0e8f-f58f-4c8f-9e30-25474e805c68_isrForCrosstalkSources_946852_35.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f2ec0e8f-f58f-4c8f-9e30-25474e805c68_isrForCrosstalkSources_946852_35.stderr parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash 531a7a1f5ed7c9149f93ef9e7aae8772 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2c799abc-3988-4fa7-94db-ecd37ff36be3_isrForCrosstalkSources_946851_16.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2c799abc-3988-4fa7-94db-ecd37ff36be3_isrForCrosstalkSources_946851_16.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash 4e9854ec679271fe80fdc356cf55a484 parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cf7568ed-8702-4f3c-b0bc-b598ec703712_isrForCrosstalkSources_946851_54.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cf7568ed-8702-4f3c-b0bc-b598ec703712_isrForCrosstalkSources_946851_54.stderr parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash 6339707911a531e9ed7b90078a69c1d8 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/653e088c-d4de-45ec-a7aa-1a229d29a9b2_isrForCrosstalkSources_946853_45.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/653e088c-d4de-45ec-a7aa-1a229d29a9b2_isrForCrosstalkSources_946853_45.stderr parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash 112c9399a0d1371df93289dc9d36e06a parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e9d1c7fc-c7e8-4d4e-b86c-838bf77f0e81_isrForCrosstalkSources_946959_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e9d1c7fc-c7e8-4d4e-b86c-838bf77f0e81_isrForCrosstalkSources_946959_42.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 4eba74e232064b5e5b9579f924b3378f parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7e5f8933-33b1-4130-804a-d17720579f95_isrForCrosstalkSources_946852_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7e5f8933-33b1-4130-804a-d17720579f95_isrForCrosstalkSources_946852_58.stderr parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash 6ee1f1dec8668351cd4183e5d5891677 parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bca5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8cd666d8-b6f3-4644-bc8d-6a875bbaeef1_isrForCrosstalkSources_946851_31.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8cd666d8-b6f3-4644-bc8d-6a875bbaeef1_isrForCrosstalkSources_946851_31.stderr parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash b0f49d241bc21b16f223039a206bf28e parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/ac2e4586-c1be-47d2-8f3e-dc0252e80282_isrForCrosstalkSources_946851_25.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/ac2e4586-c1be-47d2-8f3e-dc0252e80282_isrForCrosstalkSources_946851_25.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash e589ee6119d1c6783ee4fd671fa99798 parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c8802d80-4f2e-4d92-8806-d2f4c3ed5481_isrForCrosstalkSources_946956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c8802d80-4f2e-4d92-8806-d2f4c3ed5481_isrForCrosstalkSources_946956_18.stderr parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash dedfd1bf95e9e10ee0b91fbbb8ebf799 parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/af5741c6-68ae-424b-9df6-93d5415296c0_isrForCrosstalkSources_946957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/af5741c6-68ae-424b-9df6-93d5415296c0_isrForCrosstalkSources_946957_17.stderr parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 7c7ced307c60e84d5b046f53d779bb4e parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/dad996a0-de32-4bc2-ae17-70ebffcc5b01_isrForCrosstalkSources_946853_49.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/dad996a0-de32-4bc2-ae17-70ebffcc5b01_isrForCrosstalkSources_946853_49.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash 05e54bb56ec05b0330a85945a9b607c0 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b5d7601a-bb0c-426e-a862-349b5dbd2db2_isrForCrosstalkSources_946956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b5d7601a-bb0c-426e-a862-349b5dbd2db2_isrForCrosstalkSources_946956_7.stderr parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash e300de4908a8e0b6ab51d30f4cc92582 parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/71c2beff-47cf-426d-997f-67b323b965ce_isrForCrosstalkSources_946958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/71c2beff-47cf-426d-997f-67b323b965ce_isrForCrosstalkSources_946958_52.stderr parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash 4b3072b29b4b39d023a59a2317f1ff3f parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bc99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/916c936d-6d26-4e83-818c-79c739323d1b_isrForCrosstalkSources_946958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/916c936d-6d26-4e83-818c-79c739323d1b_isrForCrosstalkSources_946958_14.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 5ec2a25c20e3e1537683d02e0d5a690d parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6e97c6ac-e8ef-4fda-ac47-7d47f162426c_isrForCrosstalkSources_946956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6e97c6ac-e8ef-4fda-ac47-7d47f162426c_isrForCrosstalkSources_946956_13.stderr parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 3e925f481d1ac80abc9ad0674ed9dcaa parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c180e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/dae5997e-2192-453d-92b0-d9ab1f38ef3c_isrForCrosstalkSources_946852_25.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/dae5997e-2192-453d-92b0-d9ab1f38ef3c_isrForCrosstalkSources_946852_25.stderr parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash 3b6eb3add0b489ca520d9ba6d318ccad parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a105c537-e93b-4654-a7a5-87246daeb6a0_isrForCrosstalkSources_946958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a105c537-e93b-4654-a7a5-87246daeb6a0_isrForCrosstalkSources_946958_50.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 564b24c43252fc0d126134b67a26c836 parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c182c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/672db64a-67a6-4f4d-87b6-60f02e9bd6e9_isrForCrosstalkSources_946957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/672db64a-67a6-4f4d-87b6-60f02e9bd6e9_isrForCrosstalkSources_946957_56.stderr parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 83 has memoization hash 9cf8b7431b5381009cf834dffc299cce parsl.dataflow.memoization INFO: Task 83 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 83 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2695ffa7-ffce-41f3-8aaa-27c7802bd1cb_isrForCrosstalkSources_946959_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2695ffa7-ffce-41f3-8aaa-27c7802bd1cb_isrForCrosstalkSources_946959_48.stderr parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 90af4553b76ea2935d4069570752adab parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c184a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a3654626-6b64-49d1-a025-1c6f260f2abc_isrForCrosstalkSources_946956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a3654626-6b64-49d1-a025-1c6f260f2abc_isrForCrosstalkSources_946956_39.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash ea7c005426a39e9d23d1ebef2cc3f376 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3c09b9a4-4834-4fe2-badd-65d998bdea4b_isrForCrosstalkSources_946958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3c09b9a4-4834-4fe2-badd-65d998bdea4b_isrForCrosstalkSources_946958_5.stderr parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 9e02018dd4264fb162e9707dc28f2830 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/36809492-884f-477c-8732-8e7674e7ed87_isrForCrosstalkSources_946853_29.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/36809492-884f-477c-8732-8e7674e7ed87_isrForCrosstalkSources_946853_29.stderr parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 1aa0b8b88ca62528dc53bbdb1cd78598 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c185e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/fadc270f-f639-473b-b796-37ae677aef6f_isrForCrosstalkSources_946959_16.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/fadc270f-f639-473b-b796-37ae677aef6f_isrForCrosstalkSources_946959_16.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 0f8e9aa0c629d25eb383ddce41a4d099 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2fdaec1f-7c4a-404e-bd5d-f720ee811f0b_isrForCrosstalkSources_946851_21.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2fdaec1f-7c4a-404e-bd5d-f720ee811f0b_isrForCrosstalkSources_946851_21.stderr parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash 70cb55c8dbc0a6adba147ec4048295bc parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c187c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/24e09187-4748-4ab5-a5b0-88617fa70be9_isrForCrosstalkSources_946959_55.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/24e09187-4748-4ab5-a5b0-88617fa70be9_isrForCrosstalkSources_946959_55.stderr parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 00f6d5f19ff1da5ed867533fcd7e6393 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/457b7c31-f36e-47e2-bf45-215d6aa0f278_isrForCrosstalkSources_946956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/457b7c31-f36e-47e2-bf45-215d6aa0f278_isrForCrosstalkSources_946956_51.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash 72ffd1b08feb0d93dc5704017744a625 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8a2a55bb-a422-4872-b7bf-83d59927c5ea_isrForCrosstalkSources_946959_60.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8a2a55bb-a422-4872-b7bf-83d59927c5ea_isrForCrosstalkSources_946959_60.stderr parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash b5ecd42369add47ac1a37eede3340602 parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a3508f6d-174a-4c0c-9307-f32c22a13917_isrForCrosstalkSources_946853_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a3508f6d-174a-4c0c-9307-f32c22a13917_isrForCrosstalkSources_946853_2.stderr parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash 1e9aa8419461c3fc3d24ac94593696ac parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c189a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b20e565a-7c45-4461-b4c2-f9a11c56d672_isrForCrosstalkSources_946852_54.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b20e565a-7c45-4461-b4c2-f9a11c56d672_isrForCrosstalkSources_946852_54.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash a2e863e20ea7de4041b50ba07cc2b491 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4c7ecc56-7c03-43c8-ab7c-20206534c357_isrForCrosstalkSources_946852_33.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4c7ecc56-7c03-43c8-ab7c-20206534c357_isrForCrosstalkSources_946852_33.stderr parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash b1882eb363abae7f7a90b70645192ec3 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e109936b-a9a1-4df1-befa-2bbcf5c621ee_isrForCrosstalkSources_946958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e109936b-a9a1-4df1-befa-2bbcf5c621ee_isrForCrosstalkSources_946958_15.stderr parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 7098983b038eee5ce13ccdb72325d860 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/598ea790-0979-4939-93be-de95890287d4_isrForCrosstalkSources_946956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/598ea790-0979-4939-93be-de95890287d4_isrForCrosstalkSources_946956_12.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 4e0ba0554aaa7508c26ef6f597131262 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fcbe0502-4be8-41fc-a87f-5e81435d3998_isrForCrosstalkSources_946958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fcbe0502-4be8-41fc-a87f-5e81435d3998_isrForCrosstalkSources_946958_17.stderr parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash f04719832ced06efd7af360ebf4a39db parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/6e223a4d-2ac5-484b-86ad-ae9d139b4f48_isrForCrosstalkSources_946958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/6e223a4d-2ac5-484b-86ad-ae9d139b4f48_isrForCrosstalkSources_946958_22.stderr parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash d4b83ad90955c1645c0a2830e877301d parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9b16ef0a-821d-4cac-8075-4287dfa23c38_isrForCrosstalkSources_946851_8.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9b16ef0a-821d-4cac-8075-4287dfa23c38_isrForCrosstalkSources_946851_8.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash 5089023926721bedf2cf33286369273f parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d7c3bf25-b97a-42de-b2df-a9c4e00d129a_isrForCrosstalkSources_946959_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d7c3bf25-b97a-42de-b2df-a9c4e00d129a_isrForCrosstalkSources_946959_8.stderr parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 28c780347c693eac14e36e2bcce4fb6c parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/1bd0a8fc-5d70-4fc0-91f4-3c44d8056bc2_isrForCrosstalkSources_946956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/1bd0a8fc-5d70-4fc0-91f4-3c44d8056bc2_isrForCrosstalkSources_946956_50.stderr parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash 339f2063ba057d3c5938085c744bc03a parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae3dc49a-e294-4ab3-9048-50595a4e1c06_isrForCrosstalkSources_946957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae3dc49a-e294-4ab3-9048-50595a4e1c06_isrForCrosstalkSources_946957_34.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash 63be1fb67ba0a1c0913ee8355177985a parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/135694f9-4bda-463f-9c45-5e9ae1f63fd1_isrForCrosstalkSources_946959_24.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/135694f9-4bda-463f-9c45-5e9ae1f63fd1_isrForCrosstalkSources_946959_24.stderr parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash 5d6985c63655d6fa510ab602c2214613 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7badad71-61cf-4d5d-9c8f-1a8847c756c9_isrForCrosstalkSources_946851_26.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7badad71-61cf-4d5d-9c8f-1a8847c756c9_isrForCrosstalkSources_946851_26.stderr parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 521aff506df6e243086b1ec6388352f5 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/702d740b-173f-413f-b419-b4c314fec7be_isrForCrosstalkSources_946852_42.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/702d740b-173f-413f-b419-b4c314fec7be_isrForCrosstalkSources_946852_42.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash 2d95aef5caf43e70f7b540356b012cca parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c191c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/83371bf9-aca8-4258-ba6d-1ea9e73676f3_isrForCrosstalkSources_946853_51.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/83371bf9-aca8-4258-ba6d-1ea9e73676f3_isrForCrosstalkSources_946853_51.stderr parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash ede8e22165d009f8d527110bc3eedbcc parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/476ac725-3964-4827-a3f0-888db1b9508c_isrForCrosstalkSources_946958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/476ac725-3964-4827-a3f0-888db1b9508c_isrForCrosstalkSources_946958_42.stderr parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash db0337dbae049ce49c645af7dbd6c65e parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9b034887-c89f-474f-a2b6-48f0f6336f38_isrForCrosstalkSources_946959_26.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9b034887-c89f-474f-a2b6-48f0f6336f38_isrForCrosstalkSources_946959_26.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash e9d48cf30c73d32529e07681d07af177 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bd48c9ff-36c7-42c5-a8a1-8648af660385_isrForCrosstalkSources_946851_35.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bd48c9ff-36c7-42c5-a8a1-8648af660385_isrForCrosstalkSources_946851_35.stderr parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash 884eae9e3680d079897b0732b0e396bf parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c193a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/31bf661b-cf29-4487-98b2-2e12b6afed99_isrForCrosstalkSources_946851_49.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/31bf661b-cf29-4487-98b2-2e12b6afed99_isrForCrosstalkSources_946851_49.stderr parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash ec1e51ef487888a0459bb910127034da parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c194e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9fc2e32c-f3a7-494e-81c3-d779403e3f48_isrForCrosstalkSources_946851_36.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9fc2e32c-f3a7-494e-81c3-d779403e3f48_isrForCrosstalkSources_946851_36.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 36963add90de15ea5a8038d45a1ffca6 parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3b6e69e8-ba92-42ec-8b58-c1f427717275_isrForCrosstalkSources_946851_51.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3b6e69e8-ba92-42ec-8b58-c1f427717275_isrForCrosstalkSources_946851_51.stderr parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash 6038bee6c2a843be8fa0ca91f23d7250 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6a67ec7-54a2-4a65-a0e5-63894a27c698_isrForCrosstalkSources_946851_32.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6a67ec7-54a2-4a65-a0e5-63894a27c698_isrForCrosstalkSources_946851_32.stderr parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 3957f4fadf916a8e32d6bb975aed5265 parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bcf89db7-8b1e-4594-890b-229db6638d23_isrForCrosstalkSources_946852_22.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bcf89db7-8b1e-4594-890b-229db6638d23_isrForCrosstalkSources_946852_22.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 0d97896dc718b9877f8154ec3776914e parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c196c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b993e6b1-f7c8-42d3-83a7-7be1ff5a9330_isrForCrosstalkSources_946853_32.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b993e6b1-f7c8-42d3-83a7-7be1ff5a9330_isrForCrosstalkSources_946853_32.stderr parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash ddf8797cd22bf8fc5e2e120791631397 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7f5fa25-2e7b-4122-819e-7972629f00b3_isrForCrosstalkSources_946852_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7f5fa25-2e7b-4122-819e-7972629f00b3_isrForCrosstalkSources_946852_2.stderr parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 9db3ef1e71d3ecfac3e738b588a09058 parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c198a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b4883db6-80a9-4396-a7e6-f2adc3a3c520_isrForCrosstalkSources_946956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b4883db6-80a9-4396-a7e6-f2adc3a3c520_isrForCrosstalkSources_946956_47.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash b2a92dd2a24521cb727930efc544874a parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a44a17e-5948-492f-8d7a-cf4c978ab883_isrForCrosstalkSources_946851_53.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a44a17e-5948-492f-8d7a-cf4c978ab883_isrForCrosstalkSources_946851_53.stderr parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash 9ad27030e2078303226674e8cc5e56e4 parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c4550e1-4b6f-40ea-b9b3-a4e95a248aa1_isrForCrosstalkSources_946853_23.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c4550e1-4b6f-40ea-b9b3-a4e95a248aa1_isrForCrosstalkSources_946853_23.stderr parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 8119b7198dc451e38bb37919448e473d parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c199e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2ccec44f-ee83-45cb-b543-a06b11acdbad_isrForCrosstalkSources_946959_19.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2ccec44f-ee83-45cb-b543-a06b11acdbad_isrForCrosstalkSources_946959_19.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash f25dfdd88635de9139a3a0cf4ab166a7 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/31300713-97e5-4ac7-bf49-4a06f5decbcd_isrForCrosstalkSources_946959_46.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/31300713-97e5-4ac7-bf49-4a06f5decbcd_isrForCrosstalkSources_946959_46.stderr parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 4d8be62841656f2ed6b30162428b1446 parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7df9e71e-523a-443c-bdb4-973e2307e013_isrForCrosstalkSources_946956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7df9e71e-523a-443c-bdb4-973e2307e013_isrForCrosstalkSources_946956_49.stderr parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash f4a9ff4ec130fedaadee12599cefabe2 parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c3ae077-aeed-497b-a3cb-0118e9e34e35_isrForCrosstalkSources_946853_7.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c3ae077-aeed-497b-a3cb-0118e9e34e35_isrForCrosstalkSources_946853_7.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 3c10ef2d4b0d103ee884837567ea738a parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/30dd2597-deae-4013-a0d6-c5f0b781a17a_isrForCrosstalkSources_946959_31.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/30dd2597-deae-4013-a0d6-c5f0b781a17a_isrForCrosstalkSources_946959_31.stderr parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 429090b164f4adefca671d3359ad5938 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8b8ea9be-c16c-4497-a460-2a670263954b_isrForCrosstalkSources_946957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8b8ea9be-c16c-4497-a460-2a670263954b_isrForCrosstalkSources_946957_39.stderr parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash f3866b13de8b025c14fb41841b8fc3c1 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/e48ea48f-40cd-4314-9ae8-dd0b5094615b_isrForCrosstalkSources_946853_34.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/e48ea48f-40cd-4314-9ae8-dd0b5094615b_isrForCrosstalkSources_946853_34.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 77e9a8c7960647f0f49bdc8b834ae862 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e1e8568d-eb40-484f-947b-c64a5f6b3f89_isrForCrosstalkSources_946851_23.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e1e8568d-eb40-484f-947b-c64a5f6b3f89_isrForCrosstalkSources_946851_23.stderr parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash 1c3177d45bb2037f7799559baf13f865 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c19f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b6ac5e9-b479-4bf3-9c5e-01c2afddca88_isrForCrosstalkSources_946851_41.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b6ac5e9-b479-4bf3-9c5e-01c2afddca88_isrForCrosstalkSources_946851_41.stderr parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 1654fd540cdc90b8f5db919db4a40996 parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6c74aa2a-cce3-4f2d-9b99-60008cd110bb_isrForCrosstalkSources_946851_6.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6c74aa2a-cce3-4f2d-9b99-60008cd110bb_isrForCrosstalkSources_946851_6.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash a0227bed38107caafe314dafe40007a1 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9dabd97b-496b-44f6-b6e8-8e40fa64ba68_isrForCrosstalkSources_946853_8.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9dabd97b-496b-44f6-b6e8-8e40fa64ba68_isrForCrosstalkSources_946853_8.stderr parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 05f32678f5bce6186215f76f7ad77e64 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd367e53-a91f-4509-b709-6400e20399d3_isrForCrosstalkSources_946852_28.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd367e53-a91f-4509-b709-6400e20399d3_isrForCrosstalkSources_946852_28.stderr parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash f9f406c8c0e4efd9c2e8e46021086b6d parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a5672902-9d25-4126-be58-5fe8ab9c68fc_isrForCrosstalkSources_946958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a5672902-9d25-4126-be58-5fe8ab9c68fc_isrForCrosstalkSources_946958_51.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 7d59cbfd7ebe5a439cc49a0433925d4b parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/64f063dd-df79-4870-96b2-9e3f50e456be_isrForCrosstalkSources_946853_43.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/64f063dd-df79-4870-96b2-9e3f50e456be_isrForCrosstalkSources_946853_43.stderr parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash c67ade165f000698a2a4cc2082f3a89b parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4564e026-61aa-4b6f-80ea-3fdcde125ece_isrForCrosstalkSources_946958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4564e026-61aa-4b6f-80ea-3fdcde125ece_isrForCrosstalkSources_946958_24.stderr parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash 91930f9e79b907231e968b04c300421b parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9c7e3ed8-0580-45d6-aa2b-ba2629653383_isrForCrosstalkSources_946956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9c7e3ed8-0580-45d6-aa2b-ba2629653383_isrForCrosstalkSources_946956_24.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 6986eae2f55ff709457655d3fa14a4d0 parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e65cbbab-ea2d-4d17-a2e8-28850d03396d_isrForCrosstalkSources_946958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e65cbbab-ea2d-4d17-a2e8-28850d03396d_isrForCrosstalkSources_946958_53.stderr parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 395bb772cb17fee84fc394f115d755ab parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9c24173a-7b85-4953-8677-321de55e37a1_isrForCrosstalkSources_946959_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9c24173a-7b85-4953-8677-321de55e37a1_isrForCrosstalkSources_946959_6.stderr parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash 4d9f2c804736fca85ef0437424db4f5e parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/187f904d-7e96-4f7e-bce6-6602426bce6d_isrForCrosstalkSources_946957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/187f904d-7e96-4f7e-bce6-6602426bce6d_isrForCrosstalkSources_946957_47.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash fea9ff700e06a204248a7710dd262f28 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/88570778-8bde-47c6-a27f-ac125b2a6419_isrForCrosstalkSources_946957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/88570778-8bde-47c6-a27f-ac125b2a6419_isrForCrosstalkSources_946957_46.stderr parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 8dcbee32b34e4c80c348daf5ec9dba40 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3ce29ee8-932f-4374-86ff-8e292b20aebd_isrForCrosstalkSources_946851_15.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3ce29ee8-932f-4374-86ff-8e292b20aebd_isrForCrosstalkSources_946851_15.stderr parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash 583d790ce7503f924a6a68acee096fa0 parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/18f3184f-e350-43e1-89b3-842d1a4f5360_isrForCrosstalkSources_946956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/18f3184f-e350-43e1-89b3-842d1a4f5360_isrForCrosstalkSources_946956_60.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 3d317813319245c2297dcf99449aa995 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e3d70eb-583f-491e-86d2-7548fec1e176_isrForCrosstalkSources_946959_40.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e3d70eb-583f-491e-86d2-7548fec1e176_isrForCrosstalkSources_946959_40.stderr parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash 68571eb429901b1fcb3ce8e5d1ec3d40 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bfd55b9e-6345-4e51-9330-a8673138dfb8_isrForCrosstalkSources_946851_1.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bfd55b9e-6345-4e51-9330-a8673138dfb8_isrForCrosstalkSources_946851_1.stderr parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash bf8be2ded38efcf1885b15703d61f49c parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1aa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/db2eb3c8-03ff-41fe-a653-bca21971aa4c_isrForCrosstalkSources_946956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/db2eb3c8-03ff-41fe-a653-bca21971aa4c_isrForCrosstalkSources_946956_5.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash 62444c7dd772f2423f559a642b78205f parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/33ab279e-42c5-4d7f-a662-257b0f46c673_isrForCrosstalkSources_946852_21.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/33ab279e-42c5-4d7f-a662-257b0f46c673_isrForCrosstalkSources_946852_21.stderr parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 4b73fd20d2187aaaff7496c1b5fca54e parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:13 proc_lsst.multi:146 [INFO] found job 98211 in provider local proc_lsst.multi INFO: found job 98211 in provider local parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3ee5a670-48e4-485f-b69e-6eea51be6770_isrForCrosstalkSources_946958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3ee5a670-48e4-485f-b69e-6eea51be6770_isrForCrosstalkSources_946958_48.stderr parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash a13f2fb8df9615d5c853c057a46db614 parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0bcbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7f7a7394-8727-459b-9c17-f51ca29a0588_isrForCrosstalkSources_946853_22.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7f7a7394-8727-459b-9c17-f51ca29a0588_isrForCrosstalkSources_946853_22.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash 4d0a8c2cfc20f4e8619852b203a73e21 parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9d227508-7049-494c-a905-5871b60a2005_isrForCrosstalkSources_946851_60.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9d227508-7049-494c-a905-5871b60a2005_isrForCrosstalkSources_946851_60.stderr parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash 17b7e2efa5bf49d7a1014fc55c6f3328 parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a9cb1681-d8c3-43cc-b226-27c62dd4dc01_isrForCrosstalkSources_946853_9.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a9cb1681-d8c3-43cc-b226-27c62dd4dc01_isrForCrosstalkSources_946853_9.stderr parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash bcf284138a920fea069bdc14c420c78c parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ef17170a-7897-46de-b231-debbcce5cb60_isrForCrosstalkSources_946957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ef17170a-7897-46de-b231-debbcce5cb60_isrForCrosstalkSources_946957_22.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash d7364302e6588f45981e2aaaab185b13 parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dc752a24-d7c5-4960-87ad-a6c3114f2f5d_isrForCrosstalkSources_946959_41.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dc752a24-d7c5-4960-87ad-a6c3114f2f5d_isrForCrosstalkSources_946959_41.stderr parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash d5ec630f6e8c7ae10e117ef3785f002c parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/16801739-f6d7-450b-aeb2-b17e9c7b3241_isrForCrosstalkSources_946958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/16801739-f6d7-450b-aeb2-b17e9c7b3241_isrForCrosstalkSources_946958_28.stderr parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 66aeecef34ef7d5c15b2e26411f7170d parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8e1814eb-6d64-4347-85b3-08f3a1b0284d_isrForCrosstalkSources_946956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8e1814eb-6d64-4347-85b3-08f3a1b0284d_isrForCrosstalkSources_946956_44.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 96d4a90ea667b637c5cd1c9b06abd8fd parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 153 parsl.dataflow.strategy DEBUG: Executor multi has 153 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 38 more blocks parsl.executors.status_handling INFO: Scaling out by 38 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-14 04:33:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-14 04:33:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 04:33:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/30dc6f01-715a-4876-86e1-37a4d1ed4eed_isrForCrosstalkSources_946851_45.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/30dc6f01-715a-4876-86e1-37a4d1ed4eed_isrForCrosstalkSources_946851_45.stderr parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash b3f67d210a95f946b1c2718f7fc2ed23 parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fe41a5ae-af21-4246-8f33-3fad2a76b039_isrForCrosstalkSources_946957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fe41a5ae-af21-4246-8f33-3fad2a76b039_isrForCrosstalkSources_946957_2.stderr parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 732ff87581b24c6d7fb5a7f723ed1e88 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c18040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/aecea867-94d0-436c-9a30-f582ef13ef23_isrForCrosstalkSources_946851_62.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/aecea867-94d0-436c-9a30-f582ef13ef23_isrForCrosstalkSources_946851_62.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash 9d70aa2ec807027daff985b0e1947d77 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/d4e46b1f-fe15-4122-9260-85f3273f796b_isrForCrosstalkSources_946853_59.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/d4e46b1f-fe15-4122-9260-85f3273f796b_isrForCrosstalkSources_946853_59.stderr parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 860c0332b2600e4fddfaac702e46ffae parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f4101a39-75c8-413d-a9dd-80392022988c_isrForCrosstalkSources_946959_36.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f4101a39-75c8-413d-a9dd-80392022988c_isrForCrosstalkSources_946959_36.stderr parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash 04dbdbbded42e768c2bb0fa929b3e964 parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0b803735-622f-4b09-afe5-83078a636266_isrForCrosstalkSources_946853_15.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0b803735-622f-4b09-afe5-83078a636266_isrForCrosstalkSources_946853_15.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash 8930bbb63f34a0f053c96f30c4be685b parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/4c65543a-7dea-41a3-8591-44781f345d96_isrForCrosstalkSources_946957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/4c65543a-7dea-41a3-8591-44781f345d96_isrForCrosstalkSources_946957_55.stderr parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash b056d9c5c5346bfedfbd8de2b985b5b8 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/16276c1b-af19-4b23-8e4a-74849e5d49a9_isrForCrosstalkSources_946851_5.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/16276c1b-af19-4b23-8e4a-74849e5d49a9_isrForCrosstalkSources_946851_5.stderr parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash ab51c3e229decd7a1eccbe4bc3bd2a45 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/01f684cd-862a-4303-a9ad-9d70b9f252b7_isrForCrosstalkSources_946851_48.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/01f684cd-862a-4303-a9ad-9d70b9f252b7_isrForCrosstalkSources_946851_48.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash e2f4438a09e4b94f2d9dd7b5633ab0d0 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/d0b090ce-4698-4919-9904-567bfb6b59d5_isrForCrosstalkSources_946852_47.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/d0b090ce-4698-4919-9904-567bfb6b59d5_isrForCrosstalkSources_946852_47.stderr parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 39405dd26da6958a5b53e8f9a97e32f7 parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0625d398-f06a-4de3-a2d3-6bbad67fc3c0_isrForCrosstalkSources_946957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0625d398-f06a-4de3-a2d3-6bbad67fc3c0_isrForCrosstalkSources_946957_38.stderr parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash dfa9b08b78a2ab551c0220826af36568 parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b7c010fb-e94b-4475-b378-750f8d91bf0c_isrForCrosstalkSources_946851_24.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b7c010fb-e94b-4475-b378-750f8d91bf0c_isrForCrosstalkSources_946851_24.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash 1c093ba70b3f9b70e0ef8487df046879 parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/de8217ed-5025-449c-942d-e5b889cd05cd_isrForCrosstalkSources_946853_60.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/de8217ed-5025-449c-942d-e5b889cd05cd_isrForCrosstalkSources_946853_60.stderr parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 511ebebf1d2a4c9166eaf9f176cef0fc parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e25030d-a631-4ef9-8652-977b04cf6af8_isrForCrosstalkSources_946959_3.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e25030d-a631-4ef9-8652-977b04cf6af8_isrForCrosstalkSources_946959_3.stderr parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 8f09d86bb1248d65703d6045f50a543f parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f68e2767-6666-49a2-9ca7-578c8ce75d9c_isrForCrosstalkSources_946957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f68e2767-6666-49a2-9ca7-578c8ce75d9c_isrForCrosstalkSources_946957_32.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash ba48bd84a5c983110a0be6fdf5cd206c parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b5f09db4-00a9-4ed8-a245-dd9361406c04_isrForCrosstalkSources_946959_28.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b5f09db4-00a9-4ed8-a245-dd9361406c04_isrForCrosstalkSources_946959_28.stderr parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash 49901f6a6e6a9cd302ee622f974ce661 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b1149b3-3055-4b0e-97e4-114d28bf4a6b_isrForCrosstalkSources_946851_34.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b1149b3-3055-4b0e-97e4-114d28bf4a6b_isrForCrosstalkSources_946851_34.stderr parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash 40686e602d07cd04cfaa5acf8002d6ad parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb801927-f1df-4536-94f1-1014f360d118_isrForCrosstalkSources_946957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb801927-f1df-4536-94f1-1014f360d118_isrForCrosstalkSources_946957_29.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash b4e5f5cad5452d8a225136d5c3295a13 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1be20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f79ff47c-412e-49be-ac4f-c7c3e41c2ce1_isrForCrosstalkSources_946957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f79ff47c-412e-49be-ac4f-c7c3e41c2ce1_isrForCrosstalkSources_946957_14.stderr parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash d3f225cc7a91dca91f70c311c73dc4fa parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/0f0d0d0a-bd8b-4cce-97ee-8f08a348349d_isrForCrosstalkSources_946956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/0f0d0d0a-bd8b-4cce-97ee-8f08a348349d_isrForCrosstalkSources_946956_11.stderr parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash f0898ba7695d0bd5207369da1844652f parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5668d15e-7341-43d5-a965-7f78ae3d4c59_isrForCrosstalkSources_946853_56.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5668d15e-7341-43d5-a965-7f78ae3d4c59_isrForCrosstalkSources_946853_56.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 3d9b6e6a20abb99eb656710c115e70b7 parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b6705144-ca81-4727-96f4-29c22058d9b1_isrForCrosstalkSources_946851_7.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b6705144-ca81-4727-96f4-29c22058d9b1_isrForCrosstalkSources_946851_7.stderr parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 6ef5ea20d63627c0c85455d1560d3452 parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8118b22c-156d-4543-9fc5-b11958cdd0df_isrForCrosstalkSources_946957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8118b22c-156d-4543-9fc5-b11958cdd0df_isrForCrosstalkSources_946957_30.stderr parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash 7795b17aea894089bcb943af4362686f parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3055acf6-3741-4231-986d-c63bc0628189_isrForCrosstalkSources_946851_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3055acf6-3741-4231-986d-c63bc0628189_isrForCrosstalkSources_946851_58.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash a7f5b6bcc388e2c8458754967a5e24a2 parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/bd1f7d15-6d67-4062-bfcf-fb7f4a6247df_isrForCrosstalkSources_946958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/bd1f7d15-6d67-4062-bfcf-fb7f4a6247df_isrForCrosstalkSources_946958_25.stderr parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 6d1f50cf5d366c21b4d4fcc33761baea parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ca13772c-69d3-48f1-971a-6cb9caed536c_isrForCrosstalkSources_946853_48.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ca13772c-69d3-48f1-971a-6cb9caed536c_isrForCrosstalkSources_946853_48.stderr parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash 4b468c8dace85f5292286372d59c4c23 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/fe1dfb03-ce83-4bc9-9251-435be52beca4_isrForCrosstalkSources_946851_13.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/fe1dfb03-ce83-4bc9-9251-435be52beca4_isrForCrosstalkSources_946851_13.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 056ccb9e61e33a21362ed171dfd51020 parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c644a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/338b829e-7d49-4873-b4b4-e547f10f7789_isrForCrosstalkSources_946957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/338b829e-7d49-4873-b4b4-e547f10f7789_isrForCrosstalkSources_946957_19.stderr parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 5105b8eadcfafd1c95864d195e8f148f parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1c85ed03-eea1-48fb-9dca-16c4b6556356_isrForCrosstalkSources_946957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1c85ed03-eea1-48fb-9dca-16c4b6556356_isrForCrosstalkSources_946957_9.stderr parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash bfbdf91593b00eb3c2bc625a4e037775 parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/28127eb4-36d3-443c-bbfa-692094563955_isrForCrosstalkSources_946853_52.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/28127eb4-36d3-443c-bbfa-692094563955_isrForCrosstalkSources_946853_52.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash 32a2887ac9d1c8cebb1743d8b3a90876 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17030162 proc_lsst.multi INFO: [multi] job_id 17030162 2024-03-14 04:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030162 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030162 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 17030162 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-14 04:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd06bf96-ac70-4ba7-aeb7-608469cc1dd5_isrForCrosstalkSources_946852_49.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd06bf96-ac70-4ba7-aeb7-608469cc1dd5_isrForCrosstalkSources_946852_49.stderr parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 089806ee35df190af95c001ee06218fc parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/867afdf3-407b-4e67-a6bc-a5af8c55e7e7_isrForCrosstalkSources_946956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/867afdf3-407b-4e67-a6bc-a5af8c55e7e7_isrForCrosstalkSources_946956_43.stderr parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 4865a5fe8903fa8b5833fcc6281fafc7 parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a8a939-1256-4b99-bd85-8169eb6cce68_isrForCrosstalkSources_946852_12.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a8a939-1256-4b99-bd85-8169eb6cce68_isrForCrosstalkSources_946852_12.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 04ebb248b8ef230ba6a8a68b4086c106 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/358584a4-154d-4536-a01e-2e438fae818d_isrForCrosstalkSources_946851_28.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/358584a4-154d-4536-a01e-2e438fae818d_isrForCrosstalkSources_946851_28.stderr parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 8cf136f6d5ce6b3508eb73697dc2cc3f parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1c870778-e43c-4ab9-a8dc-f71c153d7d7a_isrForCrosstalkSources_946958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1c870778-e43c-4ab9-a8dc-f71c153d7d7a_isrForCrosstalkSources_946958_9.stderr parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash e1faf5d0ac3ab82fdd861b56c9ba456b parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/02eebfd4-70db-494f-931c-f6650ee258e2_isrForCrosstalkSources_946852_43.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/02eebfd4-70db-494f-931c-f6650ee258e2_isrForCrosstalkSources_946852_43.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 302927eee1fdd55f800d170f2d173b3a parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae0ef6f8-e180-422a-b710-c5a4f59899e1_isrForCrosstalkSources_946853_28.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae0ef6f8-e180-422a-b710-c5a4f59899e1_isrForCrosstalkSources_946853_28.stderr parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash c8af7c4665fa38362cbfc2d9ce6cd216 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5857b11d-ff92-454b-9a38-2051ad1fd8f1_isrForCrosstalkSources_946957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5857b11d-ff92-454b-9a38-2051ad1fd8f1_isrForCrosstalkSources_946957_10.stderr parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash ee275c6e0273175c1433d801fbd00ab2 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7d03beb3-3ce5-481b-af4a-cb93931e2267_isrForCrosstalkSources_946956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7d03beb3-3ce5-481b-af4a-cb93931e2267_isrForCrosstalkSources_946956_31.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 235834a37676cde97370220a1a9e94ff parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c679c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c7cd7a0b-f3cb-48d6-b3f8-d061115f8c7e_isrForCrosstalkSources_946957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c7cd7a0b-f3cb-48d6-b3f8-d061115f8c7e_isrForCrosstalkSources_946957_36.stderr parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 652eefb04112fa59fe7a55babe1558ff parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aa974448-c81a-4c6a-8dd7-e9bb33b3e7e6_isrForCrosstalkSources_946852_57.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aa974448-c81a-4c6a-8dd7-e9bb33b3e7e6_isrForCrosstalkSources_946852_57.stderr parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 5581e0f27b0298c231b5a448910b5068 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d60b2f19-9df4-4e85-b6f4-e7b9a3963f04_isrForCrosstalkSources_946851_57.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d60b2f19-9df4-4e85-b6f4-e7b9a3963f04_isrForCrosstalkSources_946851_57.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash f40c44f100ea9fc836a6d4b6e5d95f18 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a43ad8fd-b277-4081-a433-cd67f0399f35_isrForCrosstalkSources_946853_38.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a43ad8fd-b277-4081-a433-cd67f0399f35_isrForCrosstalkSources_946853_38.stderr parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash a10b6589269d3f6c08b6af380b94231b parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4cf1bb61-ac16-413a-bae0-fc09e811698a_isrForCrosstalkSources_946852_1.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4cf1bb61-ac16-413a-bae0-fc09e811698a_isrForCrosstalkSources_946852_1.stderr parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash 9d61f822bd037440c49aa9b1e0c8a9c6 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4b8b9054-1596-4c0d-b9ef-b7af5306caf7_isrForCrosstalkSources_946959_43.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4b8b9054-1596-4c0d-b9ef-b7af5306caf7_isrForCrosstalkSources_946959_43.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash 1c7afa0c449201fdeb4e7d4ea2f33991 parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/5b42c2e6-abe1-4a0e-bca5-33ff0e804af5_isrForCrosstalkSources_946852_53.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/5b42c2e6-abe1-4a0e-bca5-33ff0e804af5_isrForCrosstalkSources_946852_53.stderr parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash a2201a86a712e0f4a3c496fe4ca96162 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2c1eaca0-9060-480e-8f9d-56fc5c8db307_isrForCrosstalkSources_946957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2c1eaca0-9060-480e-8f9d-56fc5c8db307_isrForCrosstalkSources_946957_5.stderr parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 5bb1b3fb59b1200ffc0ccd8b25853b8b parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a57a457e-c213-4cc9-924f-68e199941d14_isrForCrosstalkSources_946851_59.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a57a457e-c213-4cc9-924f-68e199941d14_isrForCrosstalkSources_946851_59.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash 25de1bc25e8ad0e3654b5553a8508c9d parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6819116-2ac0-44ef-a734-7c6d44dcb3e5_isrForCrosstalkSources_946851_39.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6819116-2ac0-44ef-a734-7c6d44dcb3e5_isrForCrosstalkSources_946851_39.stderr parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 6c97f6d5ce0401ac11bf089c5455b2ae parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c43b06f2-0b47-49db-b4cf-4f3590119486_isrForCrosstalkSources_946853_14.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c43b06f2-0b47-49db-b4cf-4f3590119486_isrForCrosstalkSources_946853_14.stderr parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash e603bc507064ebbdfa168e8eb623f0a1 parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4611f4b2-2455-456b-b02f-395c8546082e_isrForCrosstalkSources_946852_13.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4611f4b2-2455-456b-b02f-395c8546082e_isrForCrosstalkSources_946852_13.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 51d1c7545b3cafd2fe696913f1cc85b8 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f05154e9-b7d8-4930-a0d1-9876e91bd9dd_isrForCrosstalkSources_946852_20.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f05154e9-b7d8-4930-a0d1-9876e91bd9dd_isrForCrosstalkSources_946852_20.stderr parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 694e8e6125ab4f98cc5edf64978410fe parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1da9433a-53db-44b2-8c1a-16433172ee0b_isrForCrosstalkSources_946958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1da9433a-53db-44b2-8c1a-16433172ee0b_isrForCrosstalkSources_946958_13.stderr parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 9687de6090ff87d1e2f0fb23b40e7eaf parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2b95b3a7-210a-4517-be28-a32c3470bae7_isrForCrosstalkSources_946956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2b95b3a7-210a-4517-be28-a32c3470bae7_isrForCrosstalkSources_946956_35.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 837d362956af5958b126070662d3e7f4 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/03da1849-c128-474f-ae5f-e15c5831d931_isrForCrosstalkSources_946958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/03da1849-c128-474f-ae5f-e15c5831d931_isrForCrosstalkSources_946958_4.stderr parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash eee8dde7270f118316fb76053eab15d6 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b3a0d573-b2b7-4df7-bbad-ea730418fbcb_isrForCrosstalkSources_946958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b3a0d573-b2b7-4df7-bbad-ea730418fbcb_isrForCrosstalkSources_946958_55.stderr parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash f1443fcf8d6ae3f4ef04c8375ca5e3a0 parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13bbfc2f-5dc3-40eb-a0e0-6379f56acc55_isrForCrosstalkSources_946959_12.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13bbfc2f-5dc3-40eb-a0e0-6379f56acc55_isrForCrosstalkSources_946959_12.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash b5a7a86860cacfd69bda91d68e9db966 parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/0591d93e-aeb3-40f2-8b94-26baba3d3420_isrForCrosstalkSources_946959_57.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/0591d93e-aeb3-40f2-8b94-26baba3d3420_isrForCrosstalkSources_946959_57.stderr parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 295ef31ee16f624c02c4f44b08f427f5 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6da14152-8ca2-4b12-8575-c004d1a95309_isrForCrosstalkSources_946956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6da14152-8ca2-4b12-8575-c004d1a95309_isrForCrosstalkSources_946956_54.stderr parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 89137a9a62c9b2d810171e01cf7886c3 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17030163 proc_lsst.multi INFO: [multi] job_id 17030163 2024-03-14 04:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030163 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030163 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 17030163 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-14 04:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fb1c2adf-b556-4794-a710-b99d86804ed5_isrForCrosstalkSources_946956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fb1c2adf-b556-4794-a710-b99d86804ed5_isrForCrosstalkSources_946956_59.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash ecf1fcc7eaed506a15f479442d12a147 parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/79849258-5239-42f1-9d92-9599847d3410_isrForCrosstalkSources_946956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/79849258-5239-42f1-9d92-9599847d3410_isrForCrosstalkSources_946956_22.stderr parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash f28343eeccd77f7c550ff3709258ec96 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a8f33859-b67e-4c6e-8b23-742ffc97e01f_isrForCrosstalkSources_946853_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a8f33859-b67e-4c6e-8b23-742ffc97e01f_isrForCrosstalkSources_946853_39.stderr parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash b6e2a8fd6faffe77efa307615d4568e2 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8579151f-20fe-4766-8e8d-ba819709586a_isrForCrosstalkSources_946851_56.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8579151f-20fe-4766-8e8d-ba819709586a_isrForCrosstalkSources_946851_56.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 9cca8f1ec6b66dcbf1c7967fe2c3df05 parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2b3041ca-bd0f-4855-9054-41a4bc7bc10a_isrForCrosstalkSources_946957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2b3041ca-bd0f-4855-9054-41a4bc7bc10a_isrForCrosstalkSources_946957_15.stderr parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 20288b1d8360d6d5b115dcef4b9f06ef parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9277491c-3630-4a76-a220-e4099fb3cfe8_isrForCrosstalkSources_946956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9277491c-3630-4a76-a220-e4099fb3cfe8_isrForCrosstalkSources_946956_9.stderr parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 8ea1b1f1de6ddbda8510d78171e0484d parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1b5b1e0c-d5bc-4f24-9d0d-c09645c29a9c_isrForCrosstalkSources_946957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1b5b1e0c-d5bc-4f24-9d0d-c09645c29a9c_isrForCrosstalkSources_946957_60.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 0c9e19b342a9f7173b1f8c488b106bc8 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e095400c-43a0-44d8-bf65-a61c8eb1f4c9_isrForCrosstalkSources_946852_31.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e095400c-43a0-44d8-bf65-a61c8eb1f4c9_isrForCrosstalkSources_946852_31.stderr parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash c263febe7bc60023a66c943668451f77 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/50a13037-0954-4479-9905-20eb0fbb37e7_isrForCrosstalkSources_946959_15.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/50a13037-0954-4479-9905-20eb0fbb37e7_isrForCrosstalkSources_946959_15.stderr parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash c60e30a23d942e784de3b80b305d97db parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5d870d5a-3e34-464b-bfbc-0b9ae4708957_isrForCrosstalkSources_946957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5d870d5a-3e34-464b-bfbc-0b9ae4708957_isrForCrosstalkSources_946957_31.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 8456b80672841713322b63d8c82733a2 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/9cfe5e25-ace1-4285-b080-9087dff34f79_isrForCrosstalkSources_946958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/9cfe5e25-ace1-4285-b080-9087dff34f79_isrForCrosstalkSources_946958_39.stderr parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 8f866bcb4c4eda07556885b655b127e0 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c069fe90-e79a-43cd-9545-e108ef6fb53c_isrForCrosstalkSources_946959_1.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c069fe90-e79a-43cd-9545-e108ef6fb53c_isrForCrosstalkSources_946959_1.stderr parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 30a5ce8710e16abefdc1a2a11f09a51a parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/66548884-493b-4886-95a2-dc24d0db2f99_isrForCrosstalkSources_946958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/66548884-493b-4886-95a2-dc24d0db2f99_isrForCrosstalkSources_946958_40.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 13571cda347884098550cc2a169ee676 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6cc5dc0b-fbf6-4623-86b5-2fe371089ab2_isrForCrosstalkSources_946957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6cc5dc0b-fbf6-4623-86b5-2fe371089ab2_isrForCrosstalkSources_946957_26.stderr parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash fcf889924c3ca079212bd9c010ca093f parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7a180154-c436-442a-8956-e3489db13a7e_isrForCrosstalkSources_946958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7a180154-c436-442a-8956-e3489db13a7e_isrForCrosstalkSources_946958_7.stderr parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash b40471721b9128a03fc389a7bf479613 parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1aced3a3-4172-42ee-90b2-9859ec41ac76_isrForCrosstalkSources_946959_11.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1aced3a3-4172-42ee-90b2-9859ec41ac76_isrForCrosstalkSources_946959_11.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash c8d0b82aa2702979f4162dd7ee097cfd parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3db88200-0c2c-458d-8743-613252096777_isrForCrosstalkSources_946957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3db88200-0c2c-458d-8743-613252096777_isrForCrosstalkSources_946957_23.stderr parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash e08649003493a44ff1341d8c7afc8ffa parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aeecd840-7601-4352-804a-4a3b7e07d934_isrForCrosstalkSources_946852_48.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aeecd840-7601-4352-804a-4a3b7e07d934_isrForCrosstalkSources_946852_48.stderr parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 2acaf309a2e0563ceceb8d7bfa938b0c parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f6496326-8ccc-4c88-a2cc-d6125194df76_isrForCrosstalkSources_946959_20.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f6496326-8ccc-4c88-a2cc-d6125194df76_isrForCrosstalkSources_946959_20.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 476b2b99111325e01d40058f4ff94ff6 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5e768e35-f34d-443f-b684-df554aae11e4_isrForCrosstalkSources_946956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5e768e35-f34d-443f-b684-df554aae11e4_isrForCrosstalkSources_946956_42.stderr parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 34888310810bc98585bbc0cffedeb2ce parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c660c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/846c8183-b95f-499e-a3ef-cdbd432a86e6_isrForCrosstalkSources_946852_51.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/846c8183-b95f-499e-a3ef-cdbd432a86e6_isrForCrosstalkSources_946852_51.stderr parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 1804bbcae60a82f078ffc0667ef7d99a parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/85328e7a-3c4a-4f82-a7cb-88d5fa8707c7_isrForCrosstalkSources_946957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/85328e7a-3c4a-4f82-a7cb-88d5fa8707c7_isrForCrosstalkSources_946957_48.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 0df018c4140f4b51b21f77096afb32a1 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/24320557-8dfc-4680-a8df-4b2a1888fff2_isrForCrosstalkSources_946956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/24320557-8dfc-4680-a8df-4b2a1888fff2_isrForCrosstalkSources_946956_30.stderr parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash ec3c863f2dbd0e02df8cc1d820d2f96f parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bafc38b7-c930-4085-be7c-b04b11911fa5_isrForCrosstalkSources_946957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bafc38b7-c930-4085-be7c-b04b11911fa5_isrForCrosstalkSources_946957_57.stderr parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 55e6668f740e6d3ce8e8682bb93e2a69 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1749e71b-642c-48d1-8415-8c1a3cd46578_isrForCrosstalkSources_946958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1749e71b-642c-48d1-8415-8c1a3cd46578_isrForCrosstalkSources_946958_1.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash a412c083e6032ad1e75a00f57f9d3003 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c663e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/caae8d33-7ae9-413c-966b-714c76c0f9f3_isrForCrosstalkSources_946959_35.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/caae8d33-7ae9-413c-966b-714c76c0f9f3_isrForCrosstalkSources_946959_35.stderr parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash a4ca7725a596e3178285046f43535743 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e9c55585-7a05-4264-ae07-23308e2248ad_isrForCrosstalkSources_946851_29.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e9c55585-7a05-4264-ae07-23308e2248ad_isrForCrosstalkSources_946851_29.stderr parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 8420e7992e07a68e4826fabce70a1136 parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/91650281-eeec-4ef2-b931-8bea3e2f5c5a_isrForCrosstalkSources_946956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/91650281-eeec-4ef2-b931-8bea3e2f5c5a_isrForCrosstalkSources_946956_16.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash f012c92feb32e9c6e573d0f6d77edda1 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d4ec48f0-b0cd-4b40-ac34-4f46592e59eb_isrForCrosstalkSources_946959_59.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d4ec48f0-b0cd-4b40-ac34-4f46592e59eb_isrForCrosstalkSources_946959_59.stderr parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash b7cebc889ab3b321cb0358a07413153a parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2c41d307-1ca3-4960-91bc-15d19f494124_isrForCrosstalkSources_946956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2c41d307-1ca3-4960-91bc-15d19f494124_isrForCrosstalkSources_946956_23.stderr parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash 36ca2912b1fd71bf811e7b71ade31632 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b8a25931-412c-454a-8ade-6e22125fa1b5_isrForCrosstalkSources_946958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b8a25931-412c-454a-8ade-6e22125fa1b5_isrForCrosstalkSources_946958_54.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash a49a232377a96d12158416c353f8ca3b parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a706d2c1-c08b-4d13-8956-f9082c549383_isrForCrosstalkSources_946959_53.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a706d2c1-c08b-4d13-8956-f9082c549383_isrForCrosstalkSources_946959_53.stderr parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 29a1b853aac90bf385c5858722cdf48c parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c665c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f2c8a970-73fa-401c-8ed6-91d9f4ec7b2b_isrForCrosstalkSources_946959_30.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f2c8a970-73fa-401c-8ed6-91d9f4ec7b2b_isrForCrosstalkSources_946959_30.stderr parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash dce2567cfaa7cf739004d800f021e8cb parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c2c8ed97-6651-4583-b776-615ed27b3005_isrForCrosstalkSources_946853_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c2c8ed97-6651-4583-b776-615ed27b3005_isrForCrosstalkSources_946853_24.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash 78a54c4722860af36646c108197debf6 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/67539512-d846-4669-b114-7d5a5c35b8ed_isrForCrosstalkSources_946956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/67539512-d846-4669-b114-7d5a5c35b8ed_isrForCrosstalkSources_946956_53.stderr parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash e5db94fdbbfffb4a6ecee7b0db5e1f1c parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c677e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4b908bb7-8558-48d3-a5c2-94b9c59f5dac_isrForCrosstalkSources_946852_4.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4b908bb7-8558-48d3-a5c2-94b9c59f5dac_isrForCrosstalkSources_946852_4.stderr parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 0ea548c9ef9d2a08f56b7236e7ca73ea parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c676a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/820c3f14-2ed7-4968-9a80-1de8bf8578f7_isrForCrosstalkSources_946851_9.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/820c3f14-2ed7-4968-9a80-1de8bf8578f7_isrForCrosstalkSources_946851_9.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash 036f5884adff7460b8a0da2b7a684135 parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7772cb4-1418-4fbe-8f2a-9a6f2e68de31_isrForCrosstalkSources_946852_24.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7772cb4-1418-4fbe-8f2a-9a6f2e68de31_isrForCrosstalkSources_946852_24.stderr parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash d70fb38a03efdc0c5c647320b05bd479 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4f1131c2-cc14-4ddb-ab95-a6b6b1bd1573_isrForCrosstalkSources_946959_14.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4f1131c2-cc14-4ddb-ab95-a6b6b1bd1573_isrForCrosstalkSources_946959_14.stderr parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 1979514077b4a118a271937d37df8cb9 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:14 proc_lsst.multi:170 [INFO] [multi] job_id 17030164 proc_lsst.multi INFO: [multi] job_id 17030164 2024-03-14 04:33:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030164 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030164 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 17030164 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-14 04:33:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a053c46d-c01f-4725-904c-cb0cba986b12_isrForCrosstalkSources_946956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a053c46d-c01f-4725-904c-cb0cba986b12_isrForCrosstalkSources_946956_4.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 75d1c5733515be263c3a7532d3b5c53b parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5d94eced-766a-430a-9a21-67921674a3fb_isrForCrosstalkSources_946956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5d94eced-766a-430a-9a21-67921674a3fb_isrForCrosstalkSources_946956_36.stderr parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash 42a72197d0a65fa2099511d1b5a17b9a parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c7591870-5939-4e51-ab7f-41196c6161a8_isrForCrosstalkSources_946852_30.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c7591870-5939-4e51-ab7f-41196c6161a8_isrForCrosstalkSources_946852_30.stderr parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 973db377d346dc3ec06f21f36f5aaca0 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/08c4a4d3-26e5-424d-b58c-7f084a2819fc_isrForCrosstalkSources_946959_32.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/08c4a4d3-26e5-424d-b58c-7f084a2819fc_isrForCrosstalkSources_946959_32.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash f7d9681a383b74039ff08ea6d8548fc3 parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c1b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5ea25877-9b7c-46ff-a5bc-3c318879bc61_isrForCrosstalkSources_946853_25.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5ea25877-9b7c-46ff-a5bc-3c318879bc61_isrForCrosstalkSources_946853_25.stderr parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash af49b279b52a995bfdb5d2c327cb14c3 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/a319e387-e62d-4f05-a2d9-821a601b6423_isrForCrosstalkSources_946957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/a319e387-e62d-4f05-a2d9-821a601b6423_isrForCrosstalkSources_946957_35.stderr parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash b265695f7a0b0b5ae8f43f0d80b46f15 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c27a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/a1d3bfb2-e201-4ade-9278-7fb1de8f855f_isrForCrosstalkSources_946852_56.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/a1d3bfb2-e201-4ade-9278-7fb1de8f855f_isrForCrosstalkSources_946852_56.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 472564c5f7897233934e9240daf8781d parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c25c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b6740a67-2751-492f-bf09-fc53dbbb22d8_isrForCrosstalkSources_946958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b6740a67-2751-492f-bf09-fc53dbbb22d8_isrForCrosstalkSources_946958_10.stderr parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 548dd7e7e35c7ca8272a8c5119715cb6 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/19cf9db3-16d9-455e-a45a-30cbb534398d_isrForCrosstalkSources_946957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/19cf9db3-16d9-455e-a45a-30cbb534398d_isrForCrosstalkSources_946957_51.stderr parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash b1b2ec56d67eb542adccedc01220f228 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd661873-6bf2-4059-aac0-df8294a0e189_isrForCrosstalkSources_946852_19.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd661873-6bf2-4059-aac0-df8294a0e189_isrForCrosstalkSources_946852_19.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash c791819e3f95d4de1608cf7757ee1fd9 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/33ad52c6-3516-4afb-9971-f14a28e804a1_isrForCrosstalkSources_946957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/33ad52c6-3516-4afb-9971-f14a28e804a1_isrForCrosstalkSources_946957_1.stderr parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash 9769cd6b8aae75f70527143a42cad4f9 parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f2507248-db43-440d-aeac-7a99225cabde_isrForCrosstalkSources_946958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f2507248-db43-440d-aeac-7a99225cabde_isrForCrosstalkSources_946958_56.stderr parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash a0186e197353d5ac882ad6d327ff4e5d parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bce655a2-794b-4c97-9d2a-4c38ac999f9c_isrForCrosstalkSources_946852_32.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bce655a2-794b-4c97-9d2a-4c38ac999f9c_isrForCrosstalkSources_946852_32.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 3de7e90e1ff2f9bf93e9a455b844a8fd parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/db115dfd-73d3-48cb-915d-04fa70d475f9_isrForCrosstalkSources_946959_39.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/db115dfd-73d3-48cb-915d-04fa70d475f9_isrForCrosstalkSources_946959_39.stderr parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash c55605359cbeb984def7a4cd82f51178 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e21e8730-2cb7-4b9c-89a4-528655fef785_isrForCrosstalkSources_946958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e21e8730-2cb7-4b9c-89a4-528655fef785_isrForCrosstalkSources_946958_62.stderr parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash 68d7bf25624b21a7409765d957346212 parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4e6746e1-3667-4d19-9df9-6503b7bf33c2_isrForCrosstalkSources_946851_42.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4e6746e1-3667-4d19-9df9-6503b7bf33c2_isrForCrosstalkSources_946851_42.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 9b3d72e0b3ab66a2435d19e409726be7 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/76b6803b-e52f-40df-9708-c30f41948588_isrForCrosstalkSources_946851_12.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/76b6803b-e52f-40df-9708-c30f41948588_isrForCrosstalkSources_946851_12.stderr parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash af94e71433f8c7c1f75b1ed7f2353ae0 parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7749163e-999d-4b04-8a71-175b4e244462_isrForCrosstalkSources_946958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7749163e-999d-4b04-8a71-175b4e244462_isrForCrosstalkSources_946958_11.stderr parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash f29727a06706ee37244a3439bee6b0a8 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4135553a-e573-4be6-983a-22604b888d62_isrForCrosstalkSources_946851_20.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4135553a-e573-4be6-983a-22604b888d62_isrForCrosstalkSources_946851_20.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash e7acc6b56dd80fa2b069f3754ab1b258 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/37fac77d-b146-4d9b-b57c-e66d2b6997be_isrForCrosstalkSources_946956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/37fac77d-b146-4d9b-b57c-e66d2b6997be_isrForCrosstalkSources_946956_3.stderr parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash f1f3be6cbf09903dfa8eb42ab5828013 parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/64d3d327-ddc2-4239-83f6-ade4745972c5_isrForCrosstalkSources_946958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/64d3d327-ddc2-4239-83f6-ade4745972c5_isrForCrosstalkSources_946958_38.stderr parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 4624efff7a60d18d6102e51a771fedc2 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/376efe91-a439-4e3d-96ec-697ccf0562b7_isrForCrosstalkSources_946956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/376efe91-a439-4e3d-96ec-697ccf0562b7_isrForCrosstalkSources_946956_58.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash f19ef819da7eca3b0460fb1a1bff5f7e parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b4e3de86-d011-445d-82e8-39631658d999_isrForCrosstalkSources_946957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b4e3de86-d011-445d-82e8-39631658d999_isrForCrosstalkSources_946957_18.stderr parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash 9412892479cb66522d5a4128bf7cd16e parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/715afa1f-edd7-4fea-b1f2-bcabe0cf0545_isrForCrosstalkSources_946958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/715afa1f-edd7-4fea-b1f2-bcabe0cf0545_isrForCrosstalkSources_946958_47.stderr parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash 497efb4518f6b94fdb90170ad29d9775 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f4ddb1e7-83a7-4f19-b552-f1e150a753ad_isrForCrosstalkSources_946958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f4ddb1e7-83a7-4f19-b552-f1e150a753ad_isrForCrosstalkSources_946958_3.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash d7301588d1a536f23f13fdd42199003f parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5577ee81-0a74-49cb-af68-0f752079f647_isrForCrosstalkSources_946853_42.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5577ee81-0a74-49cb-af68-0f752079f647_isrForCrosstalkSources_946853_42.stderr parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash f6b717c34ec4ddb2266849ee90227427 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c64680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/779314b8-b34d-4678-84bb-e9288be820da_isrForCrosstalkSources_946959_18.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/779314b8-b34d-4678-84bb-e9288be820da_isrForCrosstalkSources_946959_18.stderr parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash 38d621ee46c0934fecc0e704493f9062 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7515b70b-788d-49dc-80f5-f54e7be2cdbc_isrForCrosstalkSources_946958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7515b70b-788d-49dc-80f5-f54e7be2cdbc_isrForCrosstalkSources_946958_32.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash 98709d7154344f33094c5db79b855e39 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c66b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bf126800-e39f-4ee9-a4d3-d4abf9412d40_isrForCrosstalkSources_946852_39.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bf126800-e39f-4ee9-a4d3-d4abf9412d40_isrForCrosstalkSources_946852_39.stderr parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash a1be015f6ec3618e98801bf5e84440a1 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b0888412-a589-477b-9052-ef1c3faac0b8_isrForCrosstalkSources_946853_61.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b0888412-a589-477b-9052-ef1c3faac0b8_isrForCrosstalkSources_946853_61.stderr parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 648dd18a32b4ebf2adfe6344bff67d7a parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/361e8f7c-bb42-41ac-b832-5869dd41bf51_isrForCrosstalkSources_946851_47.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/361e8f7c-bb42-41ac-b832-5869dd41bf51_isrForCrosstalkSources_946851_47.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash df53caeed8bf3222a9945eea0c8417be parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee2e9f16-0577-4540-ad2c-6a96f319da86_isrForCrosstalkSources_946853_41.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee2e9f16-0577-4540-ad2c-6a96f319da86_isrForCrosstalkSources_946853_41.stderr parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 72324d367fb09a75fd413f75e0ec7251 parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c02c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3b21e03b-c6a1-4a36-9032-7ecbf531db0e_isrForCrosstalkSources_946959_13.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3b21e03b-c6a1-4a36-9032-7ecbf531db0e_isrForCrosstalkSources_946959_13.stderr parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 16609e3c57026cb789ff6903bdd91e91 parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6b87c82a-a981-44b8-8053-916033379ef7_isrForCrosstalkSources_946959_4.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6b87c82a-a981-44b8-8053-916033379ef7_isrForCrosstalkSources_946959_4.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash 22fc0e6906fe1b9efbf31458a60e65c3 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c00e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d571452a-0754-406e-a873-17e1624f812e_isrForCrosstalkSources_946956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d571452a-0754-406e-a873-17e1624f812e_isrForCrosstalkSources_946956_8.stderr parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 7d67915996759a5c104e83da11fc3b37 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/481083d8-765b-43ec-acad-a3fe63abcd63_isrForCrosstalkSources_946956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/481083d8-765b-43ec-acad-a3fe63abcd63_isrForCrosstalkSources_946956_29.stderr parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 340177aaf00dfac56f65bb03b2aa44e1 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fff9e9c8-117a-4319-96f4-cc800cb8e3d2_isrForCrosstalkSources_946853_40.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fff9e9c8-117a-4319-96f4-cc800cb8e3d2_isrForCrosstalkSources_946853_40.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 69d0400c373a9ee2c297f30afddd10a0 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c05e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/d32d636f-f29f-4149-8e4d-af367fc2625d_isrForCrosstalkSources_946958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/d32d636f-f29f-4149-8e4d-af367fc2625d_isrForCrosstalkSources_946958_29.stderr parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 728e4f19aae692ae4d17ca43100d199b parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c04a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8db70aef-3e05-4de1-87f6-960a12e6e5b4_isrForCrosstalkSources_946851_17.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8db70aef-3e05-4de1-87f6-960a12e6e5b4_isrForCrosstalkSources_946851_17.stderr parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 7092224fced587fe05d47f717f456112 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3ecbd25c-e07c-4cde-be61-96d67256fde2_isrForCrosstalkSources_946957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3ecbd25c-e07c-4cde-be61-96d67256fde2_isrForCrosstalkSources_946957_43.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash d79cf683586a94d8b539ca131c4b582f parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b26ccaf6-be69-46ab-9c82-62e4667c8769_isrForCrosstalkSources_946852_41.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b26ccaf6-be69-46ab-9c82-62e4667c8769_isrForCrosstalkSources_946852_41.stderr parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash e6809b73c925424b99ff676247c7898d parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb63fedf-22d1-4d77-ac6e-b0dc4786a101_isrForCrosstalkSources_946851_40.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb63fedf-22d1-4d77-ac6e-b0dc4786a101_isrForCrosstalkSources_946851_40.stderr parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 5818978e07b8d2a1f2be1d27cd4b613d parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8349c6e8-bcc7-43dc-825d-f368ec01a5f7_isrForCrosstalkSources_946956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8349c6e8-bcc7-43dc-825d-f368ec01a5f7_isrForCrosstalkSources_946956_26.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash 0c09e3164f8fa7a966d6793fbfd6c462 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c07c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c64c4463-f7f6-474e-920d-ba61d989b25c_isrForCrosstalkSources_946853_5.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c64c4463-f7f6-474e-920d-ba61d989b25c_isrForCrosstalkSources_946853_5.stderr parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 98d7120df0ad9371b709c251acb389e2 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7bca1028-fc6b-4fe2-9cb2-3fe76d33c5d2_isrForCrosstalkSources_946851_33.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7bca1028-fc6b-4fe2-9cb2-3fe76d33c5d2_isrForCrosstalkSources_946851_33.stderr parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 8d060eff6e79546179b9adfbc6a641ec parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/4d9c348f-f881-4292-8634-d9ae96efe863_isrForCrosstalkSources_946956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/4d9c348f-f881-4292-8634-d9ae96efe863_isrForCrosstalkSources_946956_57.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 074949f71141c856a4f455ee631fc46b parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c09a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ea11b491-3bf6-455b-960b-9832d0e979fc_isrForCrosstalkSources_946956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ea11b491-3bf6-455b-960b-9832d0e979fc_isrForCrosstalkSources_946956_32.stderr parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash f33ae314db096df62ba723762dbb11f4 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee0da6cf-8d2c-4f69-af51-3e53f43dc98c_isrForCrosstalkSources_946853_55.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee0da6cf-8d2c-4f69-af51-3e53f43dc98c_isrForCrosstalkSources_946853_55.stderr parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash e72f16f5787e96d1cc282e2300d5e800 parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/43817d7c-aec4-4952-9886-7f0504741277_isrForCrosstalkSources_946958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/43817d7c-aec4-4952-9886-7f0504741277_isrForCrosstalkSources_946958_37.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash ff28f52341268638c36e74001eba436e parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/31187d00-eb93-4b97-b396-ca9e3881f055_isrForCrosstalkSources_946956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/31187d00-eb93-4b97-b396-ca9e3881f055_isrForCrosstalkSources_946956_1.stderr parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash b7d25a61544bd573ea409ae4d82c8af4 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e3373ce6-779c-4a65-b64f-06444327b1bd_isrForCrosstalkSources_946851_14.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e3373ce6-779c-4a65-b64f-06444327b1bd_isrForCrosstalkSources_946851_14.stderr parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 8c981d3ddaadf2a44778144287912844 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/af514fc7-c351-4a3a-b45f-186093de5b25_isrForCrosstalkSources_946956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/af514fc7-c351-4a3a-b45f-186093de5b25_isrForCrosstalkSources_946956_19.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 19399defe50cbc11188a9145c86d3f98 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/07997d11-d3d3-41cc-9f36-0f5069d1f6ca_isrForCrosstalkSources_946959_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/07997d11-d3d3-41cc-9f36-0f5069d1f6ca_isrForCrosstalkSources_946959_61.stderr parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash a16b6de9cab0a6a5c584618bcf35d67f parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9553e31b-2442-48f6-b626-b3a72cf10559_isrForCrosstalkSources_946852_11.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9553e31b-2442-48f6-b626-b3a72cf10559_isrForCrosstalkSources_946852_11.stderr parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash 873e187c9ab0e62b1509ca9ad95e4e94 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/37c7f323-0ad2-4fc6-9eb6-f0f9dc42a6fc_isrForCrosstalkSources_946853_17.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/37c7f323-0ad2-4fc6-9eb6-f0f9dc42a6fc_isrForCrosstalkSources_946853_17.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash b4e975b2ce88ff58f715ef849b5fca63 parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/69a9c531-acc5-44e2-ba9b-c38ff307e616_isrForCrosstalkSources_946851_61.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/69a9c531-acc5-44e2-ba9b-c38ff307e616_isrForCrosstalkSources_946851_61.stderr parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash d45e6d54577267f00bfc23af6467eecc parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/891639aa-d46c-4bff-8d5e-aef729c2b52a_isrForCrosstalkSources_946852_52.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/891639aa-d46c-4bff-8d5e-aef729c2b52a_isrForCrosstalkSources_946852_52.stderr parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 82a673d8dbf2a3ead81cdf43aed4d2fe parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c11c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/dda0c2ca-24d3-4c2c-8893-2778b9782baf_isrForCrosstalkSources_946956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/dda0c2ca-24d3-4c2c-8893-2778b9782baf_isrForCrosstalkSources_946956_14.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 25922f8152143cc2b101a432963ddd3a parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c0f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/671fdade-0666-48f6-b9bd-5008aaf2d584_isrForCrosstalkSources_946958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/671fdade-0666-48f6-b9bd-5008aaf2d584_isrForCrosstalkSources_946958_27.stderr parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash b703ae51485f60760423dca45bb5a04c parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9374235a-1365-40f1-b367-c7eea7b9c5ae_isrForCrosstalkSources_946852_9.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9374235a-1365-40f1-b367-c7eea7b9c5ae_isrForCrosstalkSources_946852_9.stderr parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 6fd9320cb1d944da1bcdb3a4e7610fa6 parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c13a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/08dd767b-1655-4c9b-9883-83b1f5f99883_isrForCrosstalkSources_946957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/08dd767b-1655-4c9b-9883-83b1f5f99883_isrForCrosstalkSources_946957_20.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash ba1f5b46513795f698c4843529419d89 parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/67f7cb9e-61de-40cf-94f3-e002e3f966ce_isrForCrosstalkSources_946959_22.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/67f7cb9e-61de-40cf-94f3-e002e3f966ce_isrForCrosstalkSources_946959_22.stderr parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash a2dac86ab98671195ac5da711ea6805e parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c22a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb567bc9-5df9-46cc-a19d-4f2cbdd7eec5_isrForCrosstalkSources_946957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb567bc9-5df9-46cc-a19d-4f2cbdd7eec5_isrForCrosstalkSources_946957_4.stderr parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash bfeffc3bca68187cf39683fb84f2d4cd parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c23e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/eb254a2a-cdfd-489b-9457-a250873c5053_isrForCrosstalkSources_946959_5.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/eb254a2a-cdfd-489b-9457-a250873c5053_isrForCrosstalkSources_946959_5.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 5dfcaa6dd93e47f36d99689f50c14c3c parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2cefba49-f490-4161-b569-28f9ee35ca5d_isrForCrosstalkSources_946853_50.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2cefba49-f490-4161-b569-28f9ee35ca5d_isrForCrosstalkSources_946853_50.stderr parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash 227524316b877f583da0ee3f6666b318 parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/048c10cc-c0ec-4721-9c4f-694d9475c468_isrForCrosstalkSources_946957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/048c10cc-c0ec-4721-9c4f-694d9475c468_isrForCrosstalkSources_946957_27.stderr parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 532c99b3142757a9bd083c4d15d539f3 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c20c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17030165 proc_lsst.multi INFO: [multi] job_id 17030165 2024-03-14 04:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030165 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030165 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 17030165 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-14 04:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b8ea3e63-7512-40ed-8d45-da2a9b1b55ef_isrForCrosstalkSources_946853_27.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b8ea3e63-7512-40ed-8d45-da2a9b1b55ef_isrForCrosstalkSources_946853_27.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash 2187829f264d6df71632c79b47bc5383 parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d0423ace-40ad-4c80-92af-67d06e77f151_isrForCrosstalkSources_946959_33.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d0423ace-40ad-4c80-92af-67d06e77f151_isrForCrosstalkSources_946959_33.stderr parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash f8fb0bce0621f4293cacddc1cdc0e17d parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c67e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/12d3bc93-b88f-4619-97c7-e852d2933cc2_isrForCrosstalkSources_946958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/12d3bc93-b88f-4619-97c7-e852d2933cc2_isrForCrosstalkSources_946958_58.stderr parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash 3a9bd60fe8a8cf059b3e8f972fba23a2 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0c65a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/98a56cf1-a272-4ed1-8063-ae221be1deea_isrForCrosstalkSources_946853_19.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/98a56cf1-a272-4ed1-8063-ae221be1deea_isrForCrosstalkSources_946853_19.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 26c2f01b417a272b85400960894723eb parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6ff31201-4a57-4cd4-9be1-72ddd013b6e4_isrForCrosstalkSources_946959_23.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6ff31201-4a57-4cd4-9be1-72ddd013b6e4_isrForCrosstalkSources_946959_23.stderr parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash 22bc6288f538db5162d5bfbdcc051f75 parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/dadfc78e-9c35-4178-a494-c45015a2a1aa_isrForCrosstalkSources_946957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/dadfc78e-9c35-4178-a494-c45015a2a1aa_isrForCrosstalkSources_946957_21.stderr parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash 1b7bda97db8f59c63c8f2b3849f28525 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7094e88e-0e75-4d7f-bab0-43a83146198f_isrForCrosstalkSources_946956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7094e88e-0e75-4d7f-bab0-43a83146198f_isrForCrosstalkSources_946956_33.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash a03239f471ce8125f40e804fc9cbe7a4 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/962b5253-cb76-4a5f-94f5-78f3d13b528c_isrForCrosstalkSources_946851_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/962b5253-cb76-4a5f-94f5-78f3d13b528c_isrForCrosstalkSources_946851_18.stderr parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 0da2688ba4aaeed020cc88cc0d19128e parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/50c8144e-4c03-4ffe-8e99-ecff6c99b5a8_isrForCrosstalkSources_946852_46.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/50c8144e-4c03-4ffe-8e99-ecff6c99b5a8_isrForCrosstalkSources_946852_46.stderr parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash b12f344e68aa73acf4c56ee8ddc8cfb2 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae48b2a1-4cf3-4526-b267-c6aca056ac56_isrForCrosstalkSources_946957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae48b2a1-4cf3-4526-b267-c6aca056ac56_isrForCrosstalkSources_946957_52.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash ec76d0410753df960f6a034fc3dd975c parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c19e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae003319-37cb-459a-be5c-f2d4681150b9_isrForCrosstalkSources_946853_44.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae003319-37cb-459a-be5c-f2d4681150b9_isrForCrosstalkSources_946853_44.stderr parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash 145ac4a8b203cac93ce37ef39c5074fa parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/90c27d9d-a5fe-4502-ab5c-20fca9889582_isrForCrosstalkSources_946959_51.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/90c27d9d-a5fe-4502-ab5c-20fca9889582_isrForCrosstalkSources_946959_51.stderr parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 856fc65f108b155b3a6ed61587047ad8 parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c18a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/11b7bb77-ab60-46a4-9d9f-21be4155b093_isrForCrosstalkSources_946957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/11b7bb77-ab60-46a4-9d9f-21be4155b093_isrForCrosstalkSources_946957_45.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash bda8d2d7836125be8a50a716b3efcf8d parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f37b2b9a-c483-484e-88da-2c5dda365b13_isrForCrosstalkSources_946958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f37b2b9a-c483-484e-88da-2c5dda365b13_isrForCrosstalkSources_946958_59.stderr parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 6adef19ed8d818f05898f458ba63f0fd parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/f88ad868-5605-4648-8efc-c492356ce349_isrForCrosstalkSources_946851_55.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/f88ad868-5605-4648-8efc-c492356ce349_isrForCrosstalkSources_946851_55.stderr parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash aa68ea0dd8916c72d75325713e8d226e parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c16c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0a9d6551-f4b8-489b-be1b-29798d2918eb_isrForCrosstalkSources_946957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0a9d6551-f4b8-489b-be1b-29798d2918eb_isrForCrosstalkSources_946957_58.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 83d005edd200b4005120a985f9ec1966 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8e1673ad-f793-4164-8c79-ebaccc5e9a85_isrForCrosstalkSources_946959_52.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8e1673ad-f793-4164-8c79-ebaccc5e9a85_isrForCrosstalkSources_946959_52.stderr parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 726e103e1cd7c6bff6add802ec21a15c parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bada5e64-3cf5-43d7-a366-c8811ce316f6_isrForCrosstalkSources_946852_27.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bada5e64-3cf5-43d7-a366-c8811ce316f6_isrForCrosstalkSources_946852_27.stderr parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash c9f337def735a47990661a4d0ec9e3e6 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/110d54ec-deee-4a4b-b267-254f6eb2fb83_isrForCrosstalkSources_946956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/110d54ec-deee-4a4b-b267-254f6eb2fb83_isrForCrosstalkSources_946956_55.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 4c2b5a2222db453d4afb41c66c289c42 parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c14e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1462ca36-2da0-4205-968e-ce63c8a66dbc_isrForCrosstalkSources_946959_62.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1462ca36-2da0-4205-968e-ce63c8a66dbc_isrForCrosstalkSources_946959_62.stderr parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash b4f584fc470402df20f461cef7599593 parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e09466e3-2bdb-4bd2-a279-f029b35d5f21_isrForCrosstalkSources_946957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e09466e3-2bdb-4bd2-a279-f029b35d5f21_isrForCrosstalkSources_946957_25.stderr parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash b8cf3f319e5d4418e17f437dc5215d3c parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/82ba8888-76e6-4af8-ab06-8ad90d24760d_isrForCrosstalkSources_946851_22.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/82ba8888-76e6-4af8-ab06-8ad90d24760d_isrForCrosstalkSources_946851_22.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash 6c8c37381338c25381e59d6e8c07c9d6 parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4658087a-495c-4ff0-b460-76d66e0d5e65_isrForCrosstalkSources_946958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4658087a-495c-4ff0-b460-76d66e0d5e65_isrForCrosstalkSources_946958_31.stderr parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash c9db31740f2be12b6995ca1354a22d52 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c28e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/087b00ad-9885-49bb-bf9c-55189384c020_isrForCrosstalkSources_946959_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/087b00ad-9885-49bb-bf9c-55189384c020_isrForCrosstalkSources_946959_25.stderr parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash f0649cad19cca3091250c9d78b141213 parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/5aa24502-766a-4321-a079-48617c8414e0_isrForCrosstalkSources_946959_49.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/5aa24502-766a-4321-a079-48617c8414e0_isrForCrosstalkSources_946959_49.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 23728097ef8252165f5296a6f5c60b2d parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c8f2d01d-9af9-4674-9f91-07738176ea30_isrForCrosstalkSources_946957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c8f2d01d-9af9-4674-9f91-07738176ea30_isrForCrosstalkSources_946957_61.stderr parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 9935376958d27995e276bd46be9c9e1c parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2b60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17030166 proc_lsst.multi INFO: [multi] job_id 17030166 2024-03-14 04:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030166 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030166 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 17030166 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-14 04:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9ca9789a-e574-4c1f-9a9c-cf144fcfb4e4_isrForCrosstalkSources_946959_27.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9ca9789a-e574-4c1f-9a9c-cf144fcfb4e4_isrForCrosstalkSources_946959_27.stderr parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 8fb4570dde8838ee61b345a95a6a918d parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/af7305ca-5da0-4ef4-96ea-5331c9153e3a_isrForCrosstalkSources_946852_60.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/af7305ca-5da0-4ef4-96ea-5331c9153e3a_isrForCrosstalkSources_946852_60.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 9f19e2bea7771425d961deb2a571f413 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a37ae64c-a732-4fb7-8ae0-a896b474a028_isrForCrosstalkSources_946959_54.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a37ae64c-a732-4fb7-8ae0-a896b474a028_isrForCrosstalkSources_946959_54.stderr parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash eaf964ed32350c9ad9ced7403809e8a7 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/cfc60417-d761-4b3c-b091-30da184a4bf7_isrForCrosstalkSources_946958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/cfc60417-d761-4b3c-b091-30da184a4bf7_isrForCrosstalkSources_946958_46.stderr parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 5a6504e96a3e9d7cb9af032a89307663 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c6e0ad0a-11e4-4e1b-b11f-9c0517dd2d04_isrForCrosstalkSources_946958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c6e0ad0a-11e4-4e1b-b11f-9c0517dd2d04_isrForCrosstalkSources_946958_49.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 07397781a2f3c57fc169e840641b7427 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e655ad6-6c9f-4083-81c4-b1c5e4ecf9d8_isrForCrosstalkSources_946959_7.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e655ad6-6c9f-4083-81c4-b1c5e4ecf9d8_isrForCrosstalkSources_946959_7.stderr parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash b0f64ab621aec5fcd7f994020cb9c4ec parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8b20cdcc-ee33-4d49-8377-ac9d94751bb0_isrForCrosstalkSources_946853_46.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8b20cdcc-ee33-4d49-8377-ac9d94751bb0_isrForCrosstalkSources_946853_46.stderr parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash e4ec8f489193c276ad6c901f8bde3b30 parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2d40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb31f737-d90a-4c2f-9ec9-956ccf9d4c6b_isrForCrosstalkSources_946851_4.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb31f737-d90a-4c2f-9ec9-956ccf9d4c6b_isrForCrosstalkSources_946851_4.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash b083e2381f7f1d039ddebb838fd98318 parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2e80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ab66a1d5-ce0a-4a3e-ae77-d6d362108406_isrForCrosstalkSources_946956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ab66a1d5-ce0a-4a3e-ae77-d6d362108406_isrForCrosstalkSources_946956_62.stderr parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e22e04df56313dd088088e45ceea5101 parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/10ab4408-6265-48ba-899a-eca786ece055_isrForCrosstalkSources_946958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/10ab4408-6265-48ba-899a-eca786ece055_isrForCrosstalkSources_946958_21.stderr parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash e3bd9409589178694e70c698c01ace67 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a799e0-58b3-49ef-9984-6f3a952836a1_isrForCrosstalkSources_946852_14.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a799e0-58b3-49ef-9984-6f3a952836a1_isrForCrosstalkSources_946852_14.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash 847a5888b7f39767952c6c73d7fff200 parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/55f98655-0d89-4cdb-ac45-1f100eb32744_isrForCrosstalkSources_946852_18.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/55f98655-0d89-4cdb-ac45-1f100eb32744_isrForCrosstalkSources_946852_18.stderr parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 85ae11cb759d9372e046c450cf9cb9e1 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ee1e1ce6-9205-4ef4-b7c3-73cb8d4bfe55_isrForCrosstalkSources_946957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ee1e1ce6-9205-4ef4-b7c3-73cb8d4bfe55_isrForCrosstalkSources_946957_3.stderr parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash 3e90a913db3a40094a76ebdeff133a7d parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/831da753-f95f-4b0b-88c8-093165cecdb9_isrForCrosstalkSources_946956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/831da753-f95f-4b0b-88c8-093165cecdb9_isrForCrosstalkSources_946956_10.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash e56fb8be82951647d90d2700e4024177 parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/29952163-f276-4268-af32-e62c41be5aee_isrForCrosstalkSources_946957_8.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/29952163-f276-4268-af32-e62c41be5aee_isrForCrosstalkSources_946957_8.stderr parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 5f3ba8ff6a7c3c6ef122b1de2c32b915 parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/7bc1d307-436b-498b-b7d9-3ce2502bd1c3_isrForCrosstalkSources_946957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/7bc1d307-436b-498b-b7d9-3ce2502bd1c3_isrForCrosstalkSources_946957_28.stderr parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash cba47e9f4bddff8a1535faac3a375f50 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b99bfbe9-c8a4-4314-ad0f-5e71bd22fd6d_isrForCrosstalkSources_946959_47.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b99bfbe9-c8a4-4314-ad0f-5e71bd22fd6d_isrForCrosstalkSources_946959_47.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 4d4f56af4de70919d5deac9a6d580eb2 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7e995e3d-2077-458c-8358-e749bf2501be_isrForCrosstalkSources_946956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7e995e3d-2077-458c-8358-e749bf2501be_isrForCrosstalkSources_946956_27.stderr parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash f2c8c18c10b347875f6a4a461e4acfab parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c32e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/2315a55d-3ef7-4b40-8353-ae216e29f436_isrForCrosstalkSources_946958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/2315a55d-3ef7-4b40-8353-ae216e29f436_isrForCrosstalkSources_946958_44.stderr parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash 5fd1b2c0083db5a635dcad0d45819688 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/77a6740c-5fa6-4fa9-b18b-a321d8a628d6_isrForCrosstalkSources_946958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/77a6740c-5fa6-4fa9-b18b-a321d8a628d6_isrForCrosstalkSources_946958_23.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 4a9baeb1f878595a2c64ae5af0331ab6 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7fefe133-073b-4301-9959-acd94a90ce19_isrForCrosstalkSources_946851_10.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7fefe133-073b-4301-9959-acd94a90ce19_isrForCrosstalkSources_946851_10.stderr parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash 7a10f45c79c718ccf1a361d122ff81dc parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c31a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661a8dd2-34ec-4db1-997a-e6458a4218a1_isrForCrosstalkSources_946956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661a8dd2-34ec-4db1-997a-e6458a4218a1_isrForCrosstalkSources_946956_38.stderr parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash f1e8549dcc94cd74902612374d10640f parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8ae5b373-4042-4235-ac25-a811eb5e847e_isrForCrosstalkSources_946957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8ae5b373-4042-4235-ac25-a811eb5e847e_isrForCrosstalkSources_946957_40.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 56f529f6512798e33bafcdaba9cb555f parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ca877b40-5ffd-461a-a30c-cdff1cd3fd4a_isrForCrosstalkSources_946956_56.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ca877b40-5ffd-461a-a30c-cdff1cd3fd4a_isrForCrosstalkSources_946956_56.stderr parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash a8166e2bc631b872d40fe8fa65882396 parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c34c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/785f904e-c0a7-440f-be7c-6a4120b035b7_isrForCrosstalkSources_946957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/785f904e-c0a7-440f-be7c-6a4120b035b7_isrForCrosstalkSources_946957_42.stderr parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 0ef5b5a8af86d9d677fc048d6705e7b0 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cba3c0de-bfef-459e-9079-132b53e94130_isrForCrosstalkSources_946853_47.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cba3c0de-bfef-459e-9079-132b53e94130_isrForCrosstalkSources_946853_47.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash 6efb877d0583214ecd99f706eace54b6 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/1f5e2513-9889-4d93-b5c7-5d7de14c49b4_isrForCrosstalkSources_946852_34.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/1f5e2513-9889-4d93-b5c7-5d7de14c49b4_isrForCrosstalkSources_946852_34.stderr parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 371 has memoization hash 39eeb1c83c06f48ff23f292a2d03154e parsl.dataflow.memoization INFO: Task 371 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c36a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 371 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4dca462b-0515-46d2-9a62-b1aaca840fb6_isrForCrosstalkSources_946958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4dca462b-0515-46d2-9a62-b1aaca840fb6_isrForCrosstalkSources_946958_34.stderr parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash b4918f55792cbd43fe7866825c2051c7 parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c37e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a9f84d4-aac5-41f3-8afd-6395d8785eaf_isrForCrosstalkSources_946957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a9f84d4-aac5-41f3-8afd-6395d8785eaf_isrForCrosstalkSources_946957_59.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 62ef6a52e228270b229f17f1003e317c parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c39c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/810bf6e5-a319-486e-a548-391c9677f2dd_isrForCrosstalkSources_946957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/810bf6e5-a319-486e-a548-391c9677f2dd_isrForCrosstalkSources_946957_49.stderr parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash c43551d7224d3156bd63f164f2693c71 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a4e1da3-b9db-482b-8eef-f7e87d1d3cf0_isrForCrosstalkSources_946853_53.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a4e1da3-b9db-482b-8eef-f7e87d1d3cf0_isrForCrosstalkSources_946853_53.stderr parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash f564da81b846f7273ed57e216709c7a8 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/744b4cb9-c21a-4964-a604-cfa138084b40_isrForCrosstalkSources_946852_10.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/744b4cb9-c21a-4964-a604-cfa138084b40_isrForCrosstalkSources_946852_10.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash db13ce2861c1e6e5624a985ddb59b06b parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/05c7f1eb-6e7b-4c82-986e-4e18aad0da79_isrForCrosstalkSources_946853_57.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/05c7f1eb-6e7b-4c82-986e-4e18aad0da79_isrForCrosstalkSources_946853_57.stderr parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash ecf52fcdf01b972384d0616396d2e857 parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/f35a74bb-2c60-4293-984e-38268aa281b7_isrForCrosstalkSources_946956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/f35a74bb-2c60-4293-984e-38268aa281b7_isrForCrosstalkSources_946956_28.stderr parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 0dc6f0e7db7fb4941d42fcebf9e14332 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/9507c6e7-a1d2-42e2-bd88-e5810bbae759_isrForCrosstalkSources_946957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/9507c6e7-a1d2-42e2-bd88-e5810bbae759_isrForCrosstalkSources_946957_41.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash 538906f21711d7ad825830d2abd2204d parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b021f934-9ad9-4972-9907-292d9c4bdadb_isrForCrosstalkSources_946956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b021f934-9ad9-4972-9907-292d9c4bdadb_isrForCrosstalkSources_946956_25.stderr parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 919cbd4240e8b9493b6d8759ceecca07 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b9a6545f-f17c-4245-b6e3-f197ac0c943f_isrForCrosstalkSources_946956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b9a6545f-f17c-4245-b6e3-f197ac0c943f_isrForCrosstalkSources_946956_2.stderr parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 0c47c1fcd0ae07231ca5d9eef1bc9acf parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/390db2e5-3c94-4048-bfb2-629e39c8fc49_isrForCrosstalkSources_946958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/390db2e5-3c94-4048-bfb2-629e39c8fc49_isrForCrosstalkSources_946958_36.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash 1701a18bf740536791a12a648e3a5e18 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3ce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b32b6dfd-2538-4b09-a80d-796d92b27e21_isrForCrosstalkSources_946852_15.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b32b6dfd-2538-4b09-a80d-796d92b27e21_isrForCrosstalkSources_946852_15.stderr parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash d68843c617f8275e3632a116f10f0ac3 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c1f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:15 proc_lsst.multi:170 [INFO] [multi] job_id 17030167 proc_lsst.multi INFO: [multi] job_id 17030167 2024-03-14 04:33:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030167 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030167 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 17030167 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-14 04:33:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/461f8a6f-4580-4fe9-9128-de365128a74b_isrForCrosstalkSources_946852_40.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/461f8a6f-4580-4fe9-9128-de365128a74b_isrForCrosstalkSources_946852_40.stderr parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 628d5dfad18574948c2672d1c409ff55 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0fb3b41-519d-4f0d-adbe-808589428e02_isrForCrosstalkSources_946852_55.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0fb3b41-519d-4f0d-adbe-808589428e02_isrForCrosstalkSources_946852_55.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash 1c1b9c52abbf3e35ef6a3636ffd7c5e7 parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c686bed8-e74b-4eba-b201-2aa2d29418a5_isrForCrosstalkSources_946956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c686bed8-e74b-4eba-b201-2aa2d29418a5_isrForCrosstalkSources_946956_40.stderr parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash 329e6578c2f348ae3a128313aa6689f3 parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c2340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1b711e61-8cc6-46c3-a9b6-e44694c3d88c_isrForCrosstalkSources_946958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1b711e61-8cc6-46c3-a9b6-e44694c3d88c_isrForCrosstalkSources_946958_45.stderr parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash cfdd810440c17fe6eef33b8583b97489 parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/87ef3393-ae95-4695-9ecf-0baf96a139e5_isrForCrosstalkSources_946958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/87ef3393-ae95-4695-9ecf-0baf96a139e5_isrForCrosstalkSources_946958_12.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash a2c52d506aa09b11759549d68cfc7236 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d00c3100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3683825d-2429-4cd0-adc2-377660b07024_isrForCrosstalkSources_946853_21.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3683825d-2429-4cd0-adc2-377660b07024_isrForCrosstalkSources_946853_21.stderr parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash a2bb00ee059031bd4e1b5ef139c81a5c parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a3cb6b2-832b-400a-83ea-5c569c85ce67_isrForCrosstalkSources_946851_2.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a3cb6b2-832b-400a-83ea-5c569c85ce67_isrForCrosstalkSources_946851_2.stderr parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash 34cf6c6fdd9a614cd79fca019b868a1d parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/1b9d65ef-c057-4ebe-9131-5a20e8d1f8c3_isrForCrosstalkSources_946853_16.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/1b9d65ef-c057-4ebe-9131-5a20e8d1f8c3_isrForCrosstalkSources_946853_16.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash a0bf92e73955d91a1ce2846d390b7d33 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/41488292-923f-4059-a3f6-49c5406d3be6_isrForCrosstalkSources_946852_29.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/41488292-923f-4059-a3f6-49c5406d3be6_isrForCrosstalkSources_946852_29.stderr parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash b8d0dee3797a4dabbf14105a4c00d0e3 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01480e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/974a8879-f455-4522-8cab-3cac49738d8b_isrForCrosstalkSources_946852_36.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/974a8879-f455-4522-8cab-3cac49738d8b_isrForCrosstalkSources_946852_36.stderr parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash c2e1fd353567504d97581110b5099368 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6fbffe2f-bf76-4780-99f5-c4fbded9a2aa_isrForCrosstalkSources_946959_38.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6fbffe2f-bf76-4780-99f5-c4fbded9a2aa_isrForCrosstalkSources_946959_38.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 7708189300df2ccd60ed280d4970d9de parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/69bb6e33-33f8-4bbc-b65f-1fc16965e0e3_isrForCrosstalkSources_946852_50.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/69bb6e33-33f8-4bbc-b65f-1fc16965e0e3_isrForCrosstalkSources_946852_50.stderr parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash e56036b394c825ca31599694ae61a7c6 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/1a15b574-23dd-4c3b-9eec-fcc03c198d02_isrForCrosstalkSources_946851_37.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/1a15b574-23dd-4c3b-9eec-fcc03c198d02_isrForCrosstalkSources_946851_37.stderr parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash da6d734cf5998746f29119ad8e45f9af parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01482c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f3abb5c0-1bac-45ba-88cb-d3c82b8668d5_isrForCrosstalkSources_946853_10.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f3abb5c0-1bac-45ba-88cb-d3c82b8668d5_isrForCrosstalkSources_946853_10.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash 6a48530ed60bcb69b6a19e6a691cb300 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/abc05599-cb31-4911-b954-8bd663144e37_isrForCrosstalkSources_946957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/abc05599-cb31-4911-b954-8bd663144e37_isrForCrosstalkSources_946957_44.stderr parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash 575809104738186189e31c492249f274 parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01484a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2b826b9b-da24-496a-b199-fc0d6bf90b8f_isrForCrosstalkSources_946853_1.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2b826b9b-da24-496a-b199-fc0d6bf90b8f_isrForCrosstalkSources_946853_1.stderr parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash 04ad6528038b5779fb1b9de2f5fb3684 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e0c978db-e370-4950-89f3-f179e6cba43e_isrForCrosstalkSources_946852_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e0c978db-e370-4950-89f3-f179e6cba43e_isrForCrosstalkSources_946852_17.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash 29788f66d307e1627b8d368a5a9f1f32 parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/57a58161-5029-42e5-a000-1c3364f817c8_isrForCrosstalkSources_946853_31.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/57a58161-5029-42e5-a000-1c3364f817c8_isrForCrosstalkSources_946853_31.stderr parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash cd94dc3edb7c5bdb9af3d2d7474e9392 parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/37ca2676-dd03-4daa-a9da-85b923c8d9aa_isrForCrosstalkSources_946958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/37ca2676-dd03-4daa-a9da-85b923c8d9aa_isrForCrosstalkSources_946958_8.stderr parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash bbefd754d8c47f12db0c983d80a8609e parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/fb40bd96-27dc-43b2-9edf-0664892c193d_isrForCrosstalkSources_946852_6.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/fb40bd96-27dc-43b2-9edf-0664892c193d_isrForCrosstalkSources_946852_6.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash 716defbbfee079ba01226760515984e4 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01485e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c0e0e138-14ff-4b8b-96f7-b976527e428b_isrForCrosstalkSources_946959_21.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c0e0e138-14ff-4b8b-96f7-b976527e428b_isrForCrosstalkSources_946959_21.stderr parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 66f3ba764761e4b4b673d90ed21737cd parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5318a02f-ad87-439d-b507-f116858cb21a_isrForCrosstalkSources_946957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5318a02f-ad87-439d-b507-f116858cb21a_isrForCrosstalkSources_946957_62.stderr parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 52a96061c9f0c91af5863eb28c17268d parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9dd49445-acf6-4553-b338-2a5caedadc61_isrForCrosstalkSources_946852_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9dd49445-acf6-4553-b338-2a5caedadc61_isrForCrosstalkSources_946852_16.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash bcfbe56b140d6bae3b5495ec0de8eeab parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01487c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0c7730d-9e74-4276-a540-bd125649c337_isrForCrosstalkSources_946852_37.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0c7730d-9e74-4276-a540-bd125649c337_isrForCrosstalkSources_946852_37.stderr parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 1a320090ba799001fb17daf1bf3621c9 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/907df2be-38dd-4344-baad-8fedadcbd4e2_isrForCrosstalkSources_946852_62.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/907df2be-38dd-4344-baad-8fedadcbd4e2_isrForCrosstalkSources_946852_62.stderr parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash 930d1e0d5de8d70b86c4a72eca389e0b parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e4eda809-bc5c-47b8-bf76-c68e37c7092e_isrForCrosstalkSources_946957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e4eda809-bc5c-47b8-bf76-c68e37c7092e_isrForCrosstalkSources_946957_50.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash 0048d808b17460b7f740db54b830491a parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01489a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c3a356e3-d982-4754-96ad-d71f17e4a461_isrForCrosstalkSources_946958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c3a356e3-d982-4754-96ad-d71f17e4a461_isrForCrosstalkSources_946958_60.stderr parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash 3aeca8694f4553fa27c20e8806503d8c parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4436ee44-e1b0-4b58-a36a-f01c63dc0465_isrForCrosstalkSources_946958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4436ee44-e1b0-4b58-a36a-f01c63dc0465_isrForCrosstalkSources_946958_26.stderr parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash 3596b5f871d6e1bfcd05c1892f41dac6 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8804eec0-50d8-4d09-98b4-a43a493596da_isrForCrosstalkSources_946853_6.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8804eec0-50d8-4d09-98b4-a43a493596da_isrForCrosstalkSources_946853_6.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 02ddbe59c23d748c03f8a583dfc53b16 parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0cc59197-a2a5-463c-8829-c72ba5062566_isrForCrosstalkSources_946851_19.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0cc59197-a2a5-463c-8829-c72ba5062566_isrForCrosstalkSources_946851_19.stderr parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash b0235b7facde919efafe40fafd5ffc94 parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13b1648a-d7ef-4b0a-b3e6-e06e8645b6f8_isrForCrosstalkSources_946959_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13b1648a-d7ef-4b0a-b3e6-e06e8645b6f8_isrForCrosstalkSources_946959_29.stderr parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash da597c7d3c2648eb4170562488243413 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d78fd705-2e79-445d-9ff1-9cdd69d93c39_isrForCrosstalkSources_946956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d78fd705-2e79-445d-9ff1-9cdd69d93c39_isrForCrosstalkSources_946956_37.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 855d8bd365bcb8f0cdb2a55270545824 parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/abec3ea7-d7b2-44f5-ac9a-c5b6b5bc4f83_isrForCrosstalkSources_946959_44.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/abec3ea7-d7b2-44f5-ac9a-c5b6b5bc4f83_isrForCrosstalkSources_946959_44.stderr parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash 4508b88791b7a6eb5e357bb88a814933 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0148e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1a6a8ee5-cb00-419e-94b6-de06f7b0ca6a_isrForCrosstalkSources_946958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1a6a8ee5-cb00-419e-94b6-de06f7b0ca6a_isrForCrosstalkSources_946958_35.stderr parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash e2a72ab64ec7756ec37cc81efc17612e parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/2cfd5e65-7ec1-4a38-aa2c-eadd5a81f596_isrForCrosstalkSources_946852_8.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/2cfd5e65-7ec1-4a38-aa2c-eadd5a81f596_isrForCrosstalkSources_946852_8.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 402b8d12ec47e44f592de5a23b3ad86a parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-14 04:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17030168 proc_lsst.multi INFO: [multi] job_id 17030168 2024-03-14 04:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030168 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030168 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 17030168 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-14 04:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6b07cb9c-6c80-4a07-9ead-c4e1f76b5f10_isrForCrosstalkSources_946957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6b07cb9c-6c80-4a07-9ead-c4e1f76b5f10_isrForCrosstalkSources_946957_37.stderr parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash bf3bea7952c95c862fe371acfffd4497 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9cd3647a-f39a-4205-b346-7fc64378b713_isrForCrosstalkSources_946851_43.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9cd3647a-f39a-4205-b346-7fc64378b713_isrForCrosstalkSources_946851_43.stderr parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash 8c8ca4d80f697d0579d37eb1b8d25be4 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/81b8eb58-489c-4e6d-96c9-fbc8fd453bee_isrForCrosstalkSources_946853_4.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/81b8eb58-489c-4e6d-96c9-fbc8fd453bee_isrForCrosstalkSources_946853_4.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash f1c775a53207887606db90a2449a3cd0 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661b811e-f9ce-454b-b230-d4bb4973b510_isrForCrosstalkSources_946956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661b811e-f9ce-454b-b230-d4bb4973b510_isrForCrosstalkSources_946956_46.stderr parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash e25c190c5c47daf70553586d5fc82868 parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0eddc91f-a9e6-4453-884f-22381aaf2cee_isrForCrosstalkSources_946853_62.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0eddc91f-a9e6-4453-884f-22381aaf2cee_isrForCrosstalkSources_946853_62.stderr parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash 896a3b9fbcbf30506edce8184fea16c2 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fb7b9c1a-ecff-4684-8ac7-ad8135043b33_isrForCrosstalkSources_946958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fb7b9c1a-ecff-4684-8ac7-ad8135043b33_isrForCrosstalkSources_946958_33.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash 6f2aafc404c4cd010577d1c28b306a3b parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ac705e18-1bff-4da0-ada7-67feab9cc26f_isrForCrosstalkSources_946853_13.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ac705e18-1bff-4da0-ada7-67feab9cc26f_isrForCrosstalkSources_946853_13.stderr parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 444199f930a34ad80d74ae2d82eb14ef parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/271dc04d-0d9b-4648-8e20-cac03d458562_isrForCrosstalkSources_946852_59.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/271dc04d-0d9b-4648-8e20-cac03d458562_isrForCrosstalkSources_946852_59.stderr parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash edb551bc25c3212c60143838bccb1fbb parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d014ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bbaf50a1-424f-431e-9b3f-604eb6f60e91_isrForCrosstalkSources_946957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bbaf50a1-424f-431e-9b3f-604eb6f60e91_isrForCrosstalkSources_946957_11.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash cc4977a9b66e03f751a489465af166fd parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/12fd4296-8b05-4d22-b463-4c4c95caaea1_isrForCrosstalkSources_946956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/12fd4296-8b05-4d22-b463-4c4c95caaea1_isrForCrosstalkSources_946956_17.stderr parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash 010fe0bf2b945ad8ba4d27f1c8801c03 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/130133ae-37f2-4f3f-b13e-0eeb557395c1_isrForCrosstalkSources_946852_26.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/130133ae-37f2-4f3f-b13e-0eeb557395c1_isrForCrosstalkSources_946852_26.stderr parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash 2a8b3c2d94971a0b04df741dd1e2bebf parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/feba39e2-317a-49dc-aed2-138143a4e8a2_isrForCrosstalkSources_946853_12.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/feba39e2-317a-49dc-aed2-138143a4e8a2_isrForCrosstalkSources_946853_12.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash dc18bd13b086b929e55e90666a23ca60 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01498a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e854cb07-b972-4b9b-847e-8ff295edd2be_isrForCrosstalkSources_946958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e854cb07-b972-4b9b-847e-8ff295edd2be_isrForCrosstalkSources_946958_57.stderr parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash ba10b4efc5a1e893bfbb4695d39b671e parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fd536db6-f29a-425a-85f8-ed301fe50cc2_isrForCrosstalkSources_946853_36.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fd536db6-f29a-425a-85f8-ed301fe50cc2_isrForCrosstalkSources_946853_36.stderr parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 64d917fc0966d5ec9a65a1412a1a6c82 parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d01499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/387abb9d-54a0-4cca-b482-5dab5a63e17a_isrForCrosstalkSources_946956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/387abb9d-54a0-4cca-b482-5dab5a63e17a_isrForCrosstalkSources_946956_20.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App isrForCrosstalkSources, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 85aa6f54bc1370cb4f1bf7cb671837eb parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1486d0149800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/44ea3ff5-7af9-4732-8b1b-7c182903d639_isrForCrosstalkSources_946958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/44ea3ff5-7af9-4732-8b1b-7c182903d639_isrForCrosstalkSources_946958_18.stderr 2024-03-14 04:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17030169 proc_lsst.multi INFO: [multi] job_id 17030169 2024-03-14 04:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030169 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030169 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 17030169 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-14 04:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:16 proc_lsst.multi:170 [INFO] [multi] job_id 17030170 proc_lsst.multi INFO: [multi] job_id 17030170 2024-03-14 04:33:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030170 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030170 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 17030170 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-14 04:33:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:17 proc_lsst.multi:170 [INFO] [multi] job_id 17030171 proc_lsst.multi INFO: [multi] job_id 17030171 2024-03-14 04:33:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030171 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030171 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 17030171 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-14 04:33:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:17 proc_lsst.multi:170 [INFO] [multi] job_id 17030172 proc_lsst.multi INFO: [multi] job_id 17030172 2024-03-14 04:33:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030172 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030172 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 17030172 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-14 04:33:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:17 proc_lsst.multi:170 [INFO] [multi] job_id 17030173 proc_lsst.multi INFO: [multi] job_id 17030173 2024-03-14 04:33:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030173 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030173 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 17030173 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-14 04:33:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17030174 proc_lsst.multi INFO: [multi] job_id 17030174 2024-03-14 04:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030174 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030174 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 17030174 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-14 04:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17030175 proc_lsst.multi INFO: [multi] job_id 17030175 2024-03-14 04:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030175 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030175 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 17030175 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-14 04:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17030176 proc_lsst.multi INFO: [multi] job_id 17030176 2024-03-14 04:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030176 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030176 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 17030176 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-14 04:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:18 proc_lsst.multi:170 [INFO] [multi] job_id 17030177 proc_lsst.multi INFO: [multi] job_id 17030177 2024-03-14 04:33:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030177 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030177 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 17030177 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-14 04:33:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17030178 proc_lsst.multi INFO: [multi] job_id 17030178 2024-03-14 04:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030178 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030178 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 17030178 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-14 04:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17030179 proc_lsst.multi INFO: [multi] job_id 17030179 2024-03-14 04:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030179 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030179 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 17030179 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-14 04:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17030180 proc_lsst.multi INFO: [multi] job_id 17030180 2024-03-14 04:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030180 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030180 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 17030180 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-14 04:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17030181 proc_lsst.multi INFO: [multi] job_id 17030181 2024-03-14 04:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030181 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030181 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 17030181 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-14 04:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:19 proc_lsst.multi:170 [INFO] [multi] job_id 17030182 proc_lsst.multi INFO: [multi] job_id 17030182 2024-03-14 04:33:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 04:33:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030182 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030182 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 17030182 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-14 04:33:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 04:33:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030183 proc_lsst.multi INFO: [multi] job_id 17030183 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030183 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030183 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 17030183 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030184 proc_lsst.multi INFO: [multi] job_id 17030184 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030184 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030184 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 17030184 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030185 proc_lsst.multi INFO: [multi] job_id 17030185 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030185 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030185 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 17030185 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030186 proc_lsst.multi INFO: [multi] job_id 17030186 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030186 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030186 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 17030186 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030187 proc_lsst.multi INFO: [multi] job_id 17030187 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030187 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030187 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 17030187 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030188 proc_lsst.multi INFO: [multi] job_id 17030188 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030188 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030188 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 17030188 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:20 proc_lsst.multi:170 [INFO] [multi] job_id 17030189 proc_lsst.multi INFO: [multi] job_id 17030189 2024-03-14 04:33:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 04:33:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030189 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030189 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 17030189 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-14 04:33:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 04:33:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17030190 proc_lsst.multi INFO: [multi] job_id 17030190 2024-03-14 04:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 04:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030190 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030190 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 17030190 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-14 04:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17030191 proc_lsst.multi INFO: [multi] job_id 17030191 2024-03-14 04:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:21 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 17030191 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 17030191 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 17030191 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-14 04:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:21 proc_lsst.multi:170 [INFO] [multi] job_id 17030192 proc_lsst.multi INFO: [multi] job_id 17030192 2024-03-14 04:33:21 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:21 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:21 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17030192 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17030192 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 17030192 parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-14 04:33:21 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:21 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:21 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:21 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:21 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17030193 proc_lsst.multi INFO: [multi] job_id 17030193 2024-03-14 04:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17030193 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17030193 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 17030193 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-14 04:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17030194 proc_lsst.multi INFO: [multi] job_id 17030194 2024-03-14 04:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17030194 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17030194 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 17030194 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-14 04:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:22 proc_lsst.multi:170 [INFO] [multi] job_id 17030195 proc_lsst.multi INFO: [multi] job_id 17030195 2024-03-14 04:33:22 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:22 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17030195 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17030195 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 17030195 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-14 04:33:22 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:22 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:22 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:22 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:22 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17030196 proc_lsst.multi INFO: [multi] job_id 17030196 2024-03-14 04:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 17030196 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 17030196 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 17030196 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-14 04:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17030197 proc_lsst.multi INFO: [multi] job_id 17030197 2024-03-14 04:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030197 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030197 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 17030197 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-14 04:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17030198 proc_lsst.multi INFO: [multi] job_id 17030198 2024-03-14 04:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030198 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030198 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 17030198 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-14 04:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:23 proc_lsst.multi:170 [INFO] [multi] job_id 17030199 proc_lsst.multi INFO: [multi] job_id 17030199 2024-03-14 04:33:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030199 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030199 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 17030199 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 39, active_tasks = 434 parsl.dataflow.strategy DEBUG: Executor multi has 434 active tasks, 1/38 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 39 < max_blocks 312 so scaling out parsl.dataflow.strategy DEBUG: Requesting 70 more blocks parsl.executors.status_handling INFO: Scaling out by 70 blocks parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-14 04:33:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17030200 proc_lsst.multi INFO: [multi] job_id 17030200 2024-03-14 04:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030200 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030200 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 17030200 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-14 04:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17030201 proc_lsst.multi INFO: [multi] job_id 17030201 2024-03-14 04:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030201 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030201 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 17030201 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-14 04:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17030202 proc_lsst.multi INFO: [multi] job_id 17030202 2024-03-14 04:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030202 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030202 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 17030202 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-14 04:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17030203 proc_lsst.multi INFO: [multi] job_id 17030203 2024-03-14 04:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030203 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030203 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 17030203 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-14 04:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10583cd2a99a66f4436fea01de469180 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e9290b37-5fa5-4bd0-9fc4-593e257dd945_isrForCrosstalkSources_946958_19.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e9290b37-5fa5-4bd0-9fc4-593e257dd945_isrForCrosstalkSources_946958_19.stderr parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 257e8ce23a2d87c0f7c479703086baa1 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/12f56b25-106c-4896-ad49-4861e7d17352_isrForCrosstalkSources_946959_58.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/12f56b25-106c-4896-ad49-4861e7d17352_isrForCrosstalkSources_946959_58.stderr 2024-03-14 04:33:24 proc_lsst.multi:170 [INFO] [multi] job_id 17030204 proc_lsst.multi INFO: [multi] job_id 17030204 2024-03-14 04:33:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030204 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030204 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 17030204 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-14 04:33:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17030205 proc_lsst.multi INFO: [multi] job_id 17030205 2024-03-14 04:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030205 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030205 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 17030205 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-14 04:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17030206 proc_lsst.multi INFO: [multi] job_id 17030206 2024-03-14 04:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030206 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030206 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 17030206 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-14 04:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17030207 proc_lsst.multi INFO: [multi] job_id 17030207 2024-03-14 04:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030207 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030207 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 17030207 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-14 04:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:25 proc_lsst.multi:170 [INFO] [multi] job_id 17030208 proc_lsst.multi INFO: [multi] job_id 17030208 2024-03-14 04:33:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030208 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030208 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 17030208 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-14 04:33:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17030209 proc_lsst.multi INFO: [multi] job_id 17030209 2024-03-14 04:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030209 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030209 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 17030209 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-14 04:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17030210 proc_lsst.multi INFO: [multi] job_id 17030210 2024-03-14 04:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030210 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030210 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 17030210 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-14 04:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17030211 proc_lsst.multi INFO: [multi] job_id 17030211 2024-03-14 04:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030211 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030211 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 17030211 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-14 04:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:26 proc_lsst.multi:170 [INFO] [multi] job_id 17030213 proc_lsst.multi INFO: [multi] job_id 17030213 2024-03-14 04:33:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030213 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030213 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 17030213 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-14 04:33:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17030214 proc_lsst.multi INFO: [multi] job_id 17030214 2024-03-14 04:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030214 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030214 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 17030214 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-14 04:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17030215 proc_lsst.multi INFO: [multi] job_id 17030215 2024-03-14 04:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030215 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030215 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 17030215 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-14 04:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17030216 proc_lsst.multi INFO: [multi] job_id 17030216 2024-03-14 04:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030216 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030216 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 17030216 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-14 04:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17030217 proc_lsst.multi INFO: [multi] job_id 17030217 2024-03-14 04:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 17030217 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 17030217 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 17030217 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-14 04:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:27 proc_lsst.multi:170 [INFO] [multi] job_id 17030218 proc_lsst.multi INFO: [multi] job_id 17030218 2024-03-14 04:33:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030218 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030218 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 17030218 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-14 04:33:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17030219 proc_lsst.multi INFO: [multi] job_id 17030219 2024-03-14 04:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030219 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030219 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 17030219 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-14 04:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17030220 proc_lsst.multi INFO: [multi] job_id 17030220 2024-03-14 04:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030220 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030220 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 17030220 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-14 04:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:28 proc_lsst.multi:170 [INFO] [multi] job_id 17030221 proc_lsst.multi INFO: [multi] job_id 17030221 2024-03-14 04:33:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030221 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030221 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 17030221 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-14 04:33:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-14 04:33:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030222 proc_lsst.multi INFO: [multi] job_id 17030222 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030222 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030222 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 17030222 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030223 proc_lsst.multi INFO: [multi] job_id 17030223 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030223 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030223 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 17030223 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030224 proc_lsst.multi INFO: [multi] job_id 17030224 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030224 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030224 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 17030224 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030225 proc_lsst.multi INFO: [multi] job_id 17030225 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030225 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030225 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 17030225 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030226 proc_lsst.multi INFO: [multi] job_id 17030226 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030226 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030226 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 17030226 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030227 proc_lsst.multi INFO: [multi] job_id 17030227 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030227 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030227 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 17030227 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030228 proc_lsst.multi INFO: [multi] job_id 17030228 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030228 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030228 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 17030228 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:29 proc_lsst.multi:170 [INFO] [multi] job_id 17030229 proc_lsst.multi INFO: [multi] job_id 17030229 2024-03-14 04:33:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030229 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030229 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 17030229 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-14 04:33:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-14 04:33:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17030230 proc_lsst.multi INFO: [multi] job_id 17030230 2024-03-14 04:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030230 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030230 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 17030230 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-14 04:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17030231 proc_lsst.multi INFO: [multi] job_id 17030231 2024-03-14 04:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030231 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030231 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 17030231 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-14 04:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:30 proc_lsst.multi:170 [INFO] [multi] job_id 17030232 proc_lsst.multi INFO: [multi] job_id 17030232 2024-03-14 04:33:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030232 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030232 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 17030232 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-14 04:33:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-14 04:33:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17030233 proc_lsst.multi INFO: [multi] job_id 17030233 2024-03-14 04:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030233 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030233 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 17030233 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-14 04:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17030234 proc_lsst.multi INFO: [multi] job_id 17030234 2024-03-14 04:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030234 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030234 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 17030234 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-14 04:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:31 proc_lsst.multi:170 [INFO] [multi] job_id 17030235 proc_lsst.multi INFO: [multi] job_id 17030235 2024-03-14 04:33:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030235 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030235 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 17030235 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-14 04:33:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-14 04:33:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17030236 proc_lsst.multi INFO: [multi] job_id 17030236 2024-03-14 04:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030236 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030236 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 17030236 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-14 04:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17030237 proc_lsst.multi INFO: [multi] job_id 17030237 2024-03-14 04:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030237 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030237 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 17030237 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-14 04:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:32 proc_lsst.multi:170 [INFO] [multi] job_id 17030238 proc_lsst.multi INFO: [multi] job_id 17030238 2024-03-14 04:33:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 04:33:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030238 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030238 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 17030238 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-14 04:33:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-14 04:33:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17030239 proc_lsst.multi INFO: [multi] job_id 17030239 2024-03-14 04:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 04:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030239 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030239 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 17030239 parsl.executors.status_handling INFO: Allocated block ID 78 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-78 2024-03-14 04:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-78 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=78 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17030240 proc_lsst.multi INFO: [multi] job_id 17030240 2024-03-14 04:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 04:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030240 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030240 parsl.executors.status_handling DEBUG: Launched block 78 on executor multi with job ID 17030240 parsl.executors.status_handling INFO: Allocated block ID 79 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-79 2024-03-14 04:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-79 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=79 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:33 proc_lsst.multi:170 [INFO] [multi] job_id 17030241 proc_lsst.multi INFO: [multi] job_id 17030241 2024-03-14 04:33:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 04:33:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030241 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030241 parsl.executors.status_handling DEBUG: Launched block 79 on executor multi with job ID 17030241 parsl.executors.status_handling INFO: Allocated block ID 80 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-80 2024-03-14 04:33:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-80 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-14 04:33:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=80 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ceef63fd09c9cea08a6c16c1dcb4e14d with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b22f1848-c8e5-4e62-984d-221db95874d3_isrForCrosstalkSources_946959_34.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b22f1848-c8e5-4e62-984d-221db95874d3_isrForCrosstalkSources_946959_34.stderr 2024-03-14 04:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17030242 proc_lsst.multi INFO: [multi] job_id 17030242 2024-03-14 04:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 04:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030242 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030242 parsl.executors.status_handling DEBUG: Launched block 80 on executor multi with job ID 17030242 parsl.executors.status_handling INFO: Allocated block ID 81 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-81 2024-03-14 04:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-81 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=81 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96ed087a1ad85b8eab0a3e2a17279470 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/ab0b7c5f-1879-44b2-980c-84d25ff20a5c_isrForCrosstalkSources_946958_43.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/ab0b7c5f-1879-44b2-980c-84d25ff20a5c_isrForCrosstalkSources_946958_43.stderr 2024-03-14 04:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17030243 proc_lsst.multi INFO: [multi] job_id 17030243 2024-03-14 04:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 04:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030243 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030243 parsl.executors.status_handling DEBUG: Launched block 81 on executor multi with job ID 17030243 parsl.executors.status_handling INFO: Allocated block ID 82 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-82 2024-03-14 04:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-82 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=82 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17030244 proc_lsst.multi INFO: [multi] job_id 17030244 2024-03-14 04:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 04:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030244 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030244 parsl.executors.status_handling DEBUG: Launched block 82 on executor multi with job ID 17030244 parsl.executors.status_handling INFO: Allocated block ID 83 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-83 2024-03-14 04:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-83 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=83 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:34 proc_lsst.multi:170 [INFO] [multi] job_id 17030245 proc_lsst.multi INFO: [multi] job_id 17030245 2024-03-14 04:33:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 04:33:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030245 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030245 parsl.executors.status_handling DEBUG: Launched block 83 on executor multi with job ID 17030245 parsl.executors.status_handling INFO: Allocated block ID 84 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-84 2024-03-14 04:33:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-84 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-14 04:33:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=84 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 17030246 proc_lsst.multi INFO: [multi] job_id 17030246 2024-03-14 04:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 04:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030246 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030246 parsl.executors.status_handling DEBUG: Launched block 84 on executor multi with job ID 17030246 parsl.executors.status_handling INFO: Allocated block ID 85 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-85 2024-03-14 04:33:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-85 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=85 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 17030247 proc_lsst.multi INFO: [multi] job_id 17030247 2024-03-14 04:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030247 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030247 parsl.executors.status_handling DEBUG: Launched block 85 on executor multi with job ID 17030247 parsl.executors.status_handling INFO: Allocated block ID 86 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-86 2024-03-14 04:33:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-86 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=86 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:35 proc_lsst.multi:170 [INFO] [multi] job_id 17030248 proc_lsst.multi INFO: [multi] job_id 17030248 2024-03-14 04:33:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-14 04:33:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030248 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030248 parsl.executors.status_handling DEBUG: Launched block 86 on executor multi with job ID 17030248 parsl.executors.status_handling INFO: Allocated block ID 87 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-87 2024-03-14 04:33:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-87 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 31 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 31 2024-03-14 04:33:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=87 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:36 proc_lsst.multi:170 [INFO] [multi] job_id 17030249 proc_lsst.multi INFO: [multi] job_id 17030249 2024-03-14 04:33:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-14 04:33:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030249 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030249 parsl.executors.status_handling DEBUG: Launched block 87 on executor multi with job ID 17030249 parsl.executors.status_handling INFO: Allocated block ID 88 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-88 2024-03-14 04:33:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-88 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 32 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 32 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=88 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:36 proc_lsst.multi:170 [INFO] [multi] job_id 17030250 proc_lsst.multi INFO: [multi] job_id 17030250 2024-03-14 04:33:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-14 04:33:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030250 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030250 parsl.executors.status_handling DEBUG: Launched block 88 on executor multi with job ID 17030250 parsl.executors.status_handling INFO: Allocated block ID 89 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-89 2024-03-14 04:33:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-89 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 33 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 33 2024-03-14 04:33:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=89 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17030251 proc_lsst.multi INFO: [multi] job_id 17030251 2024-03-14 04:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-14 04:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030251 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030251 parsl.executors.status_handling DEBUG: Launched block 89 on executor multi with job ID 17030251 parsl.executors.status_handling INFO: Allocated block ID 90 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-90 2024-03-14 04:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-90 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 34 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 34 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=90 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17030252 proc_lsst.multi INFO: [multi] job_id 17030252 2024-03-14 04:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-14 04:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030252 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030252 parsl.executors.status_handling DEBUG: Launched block 90 on executor multi with job ID 17030252 parsl.executors.status_handling INFO: Allocated block ID 91 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-91 2024-03-14 04:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-91 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 35 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 35 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=91 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:37 proc_lsst.multi:170 [INFO] [multi] job_id 17030253 proc_lsst.multi INFO: [multi] job_id 17030253 2024-03-14 04:33:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-14 04:33:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030253 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030253 parsl.executors.status_handling DEBUG: Launched block 91 on executor multi with job ID 17030253 parsl.executors.status_handling INFO: Allocated block ID 92 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-92 2024-03-14 04:33:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-92 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 36 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 36 2024-03-14 04:33:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=92 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:38 proc_lsst.multi:170 [INFO] [multi] job_id 17030254 proc_lsst.multi INFO: [multi] job_id 17030254 2024-03-14 04:33:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-14 04:33:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030254 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030254 parsl.executors.status_handling DEBUG: Launched block 92 on executor multi with job ID 17030254 parsl.executors.status_handling INFO: Allocated block ID 93 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-93 2024-03-14 04:33:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-93 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 37 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 37 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=93 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:38 proc_lsst.multi:170 [INFO] [multi] job_id 17030255 proc_lsst.multi INFO: [multi] job_id 17030255 2024-03-14 04:33:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-14 04:33:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030255 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030255 parsl.executors.status_handling DEBUG: Launched block 93 on executor multi with job ID 17030255 parsl.executors.status_handling INFO: Allocated block ID 94 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-94 2024-03-14 04:33:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-94 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 38 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 38 2024-03-14 04:33:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=94 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030257 proc_lsst.multi INFO: [multi] job_id 17030257 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030257 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030257 parsl.executors.status_handling DEBUG: Launched block 94 on executor multi with job ID 17030257 parsl.executors.status_handling INFO: Allocated block ID 95 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-95 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-95 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 39 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 39 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=95 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030258 proc_lsst.multi INFO: [multi] job_id 17030258 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030258 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030258 parsl.executors.status_handling DEBUG: Launched block 95 on executor multi with job ID 17030258 parsl.executors.status_handling INFO: Allocated block ID 96 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-96 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-96 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 40 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 40 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=96 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030259 proc_lsst.multi INFO: [multi] job_id 17030259 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030259 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030259 parsl.executors.status_handling DEBUG: Launched block 96 on executor multi with job ID 17030259 parsl.executors.status_handling INFO: Allocated block ID 97 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-97 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-97 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 41 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 41 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=97 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030260 proc_lsst.multi INFO: [multi] job_id 17030260 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030260 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030260 parsl.executors.status_handling DEBUG: Launched block 97 on executor multi with job ID 17030260 parsl.executors.status_handling INFO: Allocated block ID 98 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-98 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-98 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 42 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 42 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=98 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030261 proc_lsst.multi INFO: [multi] job_id 17030261 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030261 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030261 parsl.executors.status_handling DEBUG: Launched block 98 on executor multi with job ID 17030261 parsl.executors.status_handling INFO: Allocated block ID 99 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-99 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-99 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 43 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 43 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=99 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030262 proc_lsst.multi INFO: [multi] job_id 17030262 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030262 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030262 parsl.executors.status_handling DEBUG: Launched block 99 on executor multi with job ID 17030262 parsl.executors.status_handling INFO: Allocated block ID 100 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-100 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-100 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 44 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 44 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=100 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:39 proc_lsst.multi:170 [INFO] [multi] job_id 17030263 proc_lsst.multi INFO: [multi] job_id 17030263 2024-03-14 04:33:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-14 04:33:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030263 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030263 parsl.executors.status_handling DEBUG: Launched block 100 on executor multi with job ID 17030263 parsl.executors.status_handling INFO: Allocated block ID 101 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-101 2024-03-14 04:33:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-101 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:39 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 45 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 45 2024-03-14 04:33:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=101 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030264 proc_lsst.multi INFO: [multi] job_id 17030264 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030264 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030264 parsl.executors.status_handling DEBUG: Launched block 101 on executor multi with job ID 17030264 parsl.executors.status_handling INFO: Allocated block ID 102 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-102 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-102 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 46 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 46 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=102 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030265 proc_lsst.multi INFO: [multi] job_id 17030265 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030265 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030265 parsl.executors.status_handling DEBUG: Launched block 102 on executor multi with job ID 17030265 parsl.executors.status_handling INFO: Allocated block ID 103 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-103 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-103 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 47 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 47 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=103 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030266 proc_lsst.multi INFO: [multi] job_id 17030266 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030266 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030266 parsl.executors.status_handling DEBUG: Launched block 103 on executor multi with job ID 17030266 parsl.executors.status_handling INFO: Allocated block ID 104 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-104 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-104 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 48 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 48 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=104 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030267 proc_lsst.multi INFO: [multi] job_id 17030267 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030267 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030267 parsl.executors.status_handling DEBUG: Launched block 104 on executor multi with job ID 17030267 parsl.executors.status_handling INFO: Allocated block ID 105 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-105 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-105 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 49 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 49 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=105 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030268 proc_lsst.multi INFO: [multi] job_id 17030268 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030268 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030268 parsl.executors.status_handling DEBUG: Launched block 105 on executor multi with job ID 17030268 parsl.executors.status_handling INFO: Allocated block ID 106 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-106 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-106 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 50 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 50 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=106 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030269 proc_lsst.multi INFO: [multi] job_id 17030269 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030269 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030269 parsl.executors.status_handling DEBUG: Launched block 106 on executor multi with job ID 17030269 parsl.executors.status_handling INFO: Allocated block ID 107 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-107 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-107 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 51 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 51 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=107 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:40 proc_lsst.multi:170 [INFO] [multi] job_id 17030270 proc_lsst.multi INFO: [multi] job_id 17030270 2024-03-14 04:33:40 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-14 04:33:40 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030270 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030270 parsl.executors.status_handling DEBUG: Launched block 107 on executor multi with job ID 17030270 parsl.executors.status_handling INFO: Allocated block ID 108 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-108 2024-03-14 04:33:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-108 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-14 04:33:40 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-14 04:33:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 52 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 52 2024-03-14 04:33:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.129.9,198.48.92.26,n3009,169.254.95.120,10.64.65.9 -p 0 -c 1.0 -m None --poll 10 --task_port=54858 --result_port=54943 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/runinfo/000/multi --block_id=108 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-14 04:33:41 proc_lsst.multi:170 [INFO] [multi] job_id 17030271 proc_lsst.multi INFO: [multi] job_id 17030271 2024-03-14 04:33:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 53 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 53 2024-03-14 04:33:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 256 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 256 2024-03-14 04:33:41 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 17030271 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 17030271 parsl.executors.status_handling DEBUG: Launched block 108 on executor multi with job ID 17030271 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 430 parsl.dataflow.strategy DEBUG: Executor multi has 430 active tasks, 1/108 running/pending blocks, and 65 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1777f12030d103d897a6b6fefd19f8e3 with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a57fc90-ceaa-46a0-8290-e8753abb5390_isrForCrosstalkSources_946853_33.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a57fc90-ceaa-46a0-8290-e8753abb5390_isrForCrosstalkSources_946853_33.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f013a48738ba524895e1f5db20aa55 with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8d9335e9-eb0d-44df-bbfa-ce1f65df1195_isrForCrosstalkSources_946959_45.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8d9335e9-eb0d-44df-bbfa-ce1f65df1195_isrForCrosstalkSources_946959_45.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0821f4eefcaf2ca790e1bcdf3c6fa279 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/29210118-553f-46ec-a9a5-46757fba314c_isrForCrosstalkSources_946852_23.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/29210118-553f-46ec-a9a5-46757fba314c_isrForCrosstalkSources_946852_23.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 993a80726dfcfc589aaf8217ac43cf46 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2d62fd8d-d6c7-4078-8d27-fb858fda5cdf_isrForCrosstalkSources_946957_6.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2d62fd8d-d6c7-4078-8d27-fb858fda5cdf_isrForCrosstalkSources_946957_6.stderr parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af0b70ad783a7b926814bbbd98fe24b5 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fc243564-f4eb-4ed8-941b-c28e4bf941b1_isrForCrosstalkSources_946956_45.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fc243564-f4eb-4ed8-941b-c28e4bf941b1_isrForCrosstalkSources_946956_45.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4096575ad18a1ff04d0161b62df53508 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8108ee8e-9f06-41d8-8bd0-e9b3febffa27_isrForCrosstalkSources_946853_3.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8108ee8e-9f06-41d8-8bd0-e9b3febffa27_isrForCrosstalkSources_946853_3.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d476c04f8de7ddb457fbb89e3dea3087 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c6df45e4-9922-4610-979a-f3cefcae9056_isrForCrosstalkSources_946852_5.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c6df45e4-9922-4610-979a-f3cefcae9056_isrForCrosstalkSources_946852_5.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cbdf99bb80b7facf3d263f8b4214fc8 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/3a2b93fc-aae5-462e-8f33-2d8185d38724_isrForCrosstalkSources_946956_52.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/3a2b93fc-aae5-462e-8f33-2d8185d38724_isrForCrosstalkSources_946956_52.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1be1e7f248f7c97dc498bea33fa39dd4 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7d37414c-cfbe-4c75-a40d-d951cac8ecdb_isrForCrosstalkSources_946852_61.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7d37414c-cfbe-4c75-a40d-d951cac8ecdb_isrForCrosstalkSources_946852_61.stderr 2024-03-14 04:33:46 proc_lsst.multi:146 [INFO] found job 98211 in provider local proc_lsst.multi INFO: found job 98211 in provider local 2024-03-14 04:33:46 proc_lsst.multi:146 [INFO] found job 17030162 in provider astro proc_lsst.multi INFO: found job 17030162 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:46 proc_lsst.multi:146 [INFO] found job 17030163 in provider astro proc_lsst.multi INFO: found job 17030163 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41ebd1cb899579d58e5b78df6410e1a5 with result from task 36 parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:46 proc_lsst.multi:146 [INFO] found job 17030164 in provider astro proc_lsst.multi INFO: found job 17030164 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d17f1a29-074a-4285-9f3f-0aef881aec5b_isrForCrosstalkSources_946851_52.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d17f1a29-074a-4285-9f3f-0aef881aec5b_isrForCrosstalkSources_946851_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:47 proc_lsst.multi:146 [INFO] found job 17030165 in provider astro proc_lsst.multi INFO: found job 17030165 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:47 proc_lsst.multi:146 [INFO] found job 17030166 in provider astro proc_lsst.multi INFO: found job 17030166 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:47 proc_lsst.multi:146 [INFO] found job 17030167 in provider astro proc_lsst.multi INFO: found job 17030167 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:48 proc_lsst.multi:146 [INFO] found job 17030168 in provider astro proc_lsst.multi INFO: found job 17030168 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:48 proc_lsst.multi:146 [INFO] found job 17030169 in provider astro proc_lsst.multi INFO: found job 17030169 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:49 proc_lsst.multi:146 [INFO] found job 17030170 in provider astro proc_lsst.multi INFO: found job 17030170 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:49 proc_lsst.multi:146 [INFO] found job 17030171 in provider astro proc_lsst.multi INFO: found job 17030171 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:49 proc_lsst.multi:146 [INFO] found job 17030172 in provider astro proc_lsst.multi INFO: found job 17030172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:49 proc_lsst.multi:146 [INFO] found job 17030173 in provider astro proc_lsst.multi INFO: found job 17030173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:50 proc_lsst.multi:146 [INFO] found job 17030174 in provider astro proc_lsst.multi INFO: found job 17030174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:50 proc_lsst.multi:146 [INFO] found job 17030175 in provider astro proc_lsst.multi INFO: found job 17030175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:50 proc_lsst.multi:146 [INFO] found job 17030176 in provider astro proc_lsst.multi INFO: found job 17030176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:50 proc_lsst.multi:146 [INFO] found job 17030177 in provider astro proc_lsst.multi INFO: found job 17030177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:51 proc_lsst.multi:146 [INFO] found job 17030178 in provider astro proc_lsst.multi INFO: found job 17030178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6dc1ce4d0d42adc87df73b6720c959e with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e13df8af-efbb-4779-9b7d-d932680f8c96_isrForCrosstalkSources_946959_9.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e13df8af-efbb-4779-9b7d-d932680f8c96_isrForCrosstalkSources_946959_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:51 proc_lsst.multi:146 [INFO] found job 17030179 in provider astro proc_lsst.multi INFO: found job 17030179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b190fcf92d7a88f340927fb1b2ba5a32 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/23fb6e7d-31fa-4437-b1fe-6f0552b67ee0_isrForCrosstalkSources_946957_53.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/23fb6e7d-31fa-4437-b1fe-6f0552b67ee0_isrForCrosstalkSources_946957_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:52 proc_lsst.multi:146 [INFO] found job 17030180 in provider astro proc_lsst.multi INFO: found job 17030180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ca7acb213585138f7ae90fde40315bef with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e1130ede-29ae-42fd-9191-dd64e8f2b86d_isrForCrosstalkSources_946957_16.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e1130ede-29ae-42fd-9191-dd64e8f2b86d_isrForCrosstalkSources_946957_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:52 proc_lsst.multi:146 [INFO] found job 17030181 in provider astro proc_lsst.multi INFO: found job 17030181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dca793bce165e233f986967325e1a8d with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d13a24ac-4e2c-4dcf-a8c1-f18be7e671f3_isrForCrosstalkSources_946956_48.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d13a24ac-4e2c-4dcf-a8c1-f18be7e671f3_isrForCrosstalkSources_946956_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:52 proc_lsst.multi:146 [INFO] found job 17030182 in provider astro proc_lsst.multi INFO: found job 17030182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f27743dd85708ab93aeff9d4f8144016 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d5fe790a-ba26-422e-a137-75c03b068eef_isrForCrosstalkSources_946851_50.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d5fe790a-ba26-422e-a137-75c03b068eef_isrForCrosstalkSources_946851_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:53 proc_lsst.multi:146 [INFO] found job 17030183 in provider astro proc_lsst.multi INFO: found job 17030183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:53 proc_lsst.multi:146 [INFO] found job 17030184 in provider astro proc_lsst.multi INFO: found job 17030184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:53 proc_lsst.multi:146 [INFO] found job 17030185 in provider astro proc_lsst.multi INFO: found job 17030185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77e9a8c7960647f0f49bdc8b834ae862 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e1e8568d-eb40-484f-947b-c64a5f6b3f89_isrForCrosstalkSources_946851_23.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e1e8568d-eb40-484f-947b-c64a5f6b3f89_isrForCrosstalkSources_946851_23.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3177d45bb2037f7799559baf13f865 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b6ac5e9-b479-4bf3-9c5e-01c2afddca88_isrForCrosstalkSources_946851_41.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b6ac5e9-b479-4bf3-9c5e-01c2afddca88_isrForCrosstalkSources_946851_41.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1654fd540cdc90b8f5db919db4a40996 with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6c74aa2a-cce3-4f2d-9b99-60008cd110bb_isrForCrosstalkSources_946851_6.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6c74aa2a-cce3-4f2d-9b99-60008cd110bb_isrForCrosstalkSources_946851_6.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9d48cf30c73d32529e07681d07af177 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bd48c9ff-36c7-42c5-a8a1-8648af660385_isrForCrosstalkSources_946851_35.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bd48c9ff-36c7-42c5-a8a1-8648af660385_isrForCrosstalkSources_946851_35.stderr parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2a92dd2a24521cb727930efc544874a with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a44a17e-5948-492f-8d7a-cf4c978ab883_isrForCrosstalkSources_946851_53.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a44a17e-5948-492f-8d7a-cf4c978ab883_isrForCrosstalkSources_946851_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:54 proc_lsst.multi:146 [INFO] found job 17030186 in provider astro proc_lsst.multi INFO: found job 17030186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0227bed38107caafe314dafe40007a1 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9dabd97b-496b-44f6-b6e8-8e40fa64ba68_isrForCrosstalkSources_946853_8.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9dabd97b-496b-44f6-b6e8-8e40fa64ba68_isrForCrosstalkSources_946853_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:54 proc_lsst.multi:146 [INFO] found job 17030187 in provider astro proc_lsst.multi INFO: found job 17030187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:54 proc_lsst.multi:146 [INFO] found job 17030188 in provider astro proc_lsst.multi INFO: found job 17030188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39405dd26da6958a5b53e8f9a97e32f7 with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0625d398-f06a-4de3-a2d3-6bbad67fc3c0_isrForCrosstalkSources_946957_38.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0625d398-f06a-4de3-a2d3-6bbad67fc3c0_isrForCrosstalkSources_946957_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:54 proc_lsst.multi:146 [INFO] found job 17030189 in provider astro proc_lsst.multi INFO: found job 17030189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3f225cc7a91dca91f70c311c73dc4fa with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/0f0d0d0a-bd8b-4cce-97ee-8f08a348349d_isrForCrosstalkSources_946956_11.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/0f0d0d0a-bd8b-4cce-97ee-8f08a348349d_isrForCrosstalkSources_946956_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:54 proc_lsst.multi:146 [INFO] found job 17030190 in provider astro proc_lsst.multi INFO: found job 17030190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030191 in provider astro proc_lsst.multi INFO: found job 17030191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030192 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030192 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a13f2fb8df9615d5c853c057a46db614 with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7f7a7394-8727-459b-9c17-f51ca29a0588_isrForCrosstalkSources_946853_22.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7f7a7394-8727-459b-9c17-f51ca29a0588_isrForCrosstalkSources_946853_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030193 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030193 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030194 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030194 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030195 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030195 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030196 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030196 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030197 in provider escience_a40 proc_lsst.multi INFO: found job 17030197 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030198 in provider escience_a40 proc_lsst.multi INFO: found job 17030198 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:55 proc_lsst.multi:146 [INFO] found job 17030199 in provider escience_a40 proc_lsst.multi INFO: found job 17030199 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030200 in provider escience_a40 proc_lsst.multi INFO: found job 17030200 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030201 in provider escience_a40 proc_lsst.multi INFO: found job 17030201 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030202 in provider escience_a40 proc_lsst.multi INFO: found job 17030202 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf1fcc7eaed506a15f479442d12a147 with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/79849258-5239-42f1-9d92-9599847d3410_isrForCrosstalkSources_946956_22.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/79849258-5239-42f1-9d92-9599847d3410_isrForCrosstalkSources_946956_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030203 in provider escience_a40 proc_lsst.multi INFO: found job 17030203 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030204 in provider escience_a40 proc_lsst.multi INFO: found job 17030204 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030205 in provider escience_a40 proc_lsst.multi INFO: found job 17030205 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030206 in provider escience_a40 proc_lsst.multi INFO: found job 17030206 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:56 proc_lsst.multi:146 [INFO] found job 17030207 in provider escience_a40 proc_lsst.multi INFO: found job 17030207 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 531a7a1f5ed7c9149f93ef9e7aae8772 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2c799abc-3988-4fa7-94db-ecd37ff36be3_isrForCrosstalkSources_946851_16.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2c799abc-3988-4fa7-94db-ecd37ff36be3_isrForCrosstalkSources_946851_16.stderr parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6339707911a531e9ed7b90078a69c1d8 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/653e088c-d4de-45ec-a7aa-1a229d29a9b2_isrForCrosstalkSources_946853_45.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/653e088c-d4de-45ec-a7aa-1a229d29a9b2_isrForCrosstalkSources_946853_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030208 in provider escience_a40 proc_lsst.multi INFO: found job 17030208 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030209 in provider escience_a40 proc_lsst.multi INFO: found job 17030209 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030210 in provider escience_a40 proc_lsst.multi INFO: found job 17030210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030211 in provider escience_a40 proc_lsst.multi INFO: found job 17030211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9854ec679271fe80fdc356cf55a484 with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cf7568ed-8702-4f3c-b0bc-b598ec703712_isrForCrosstalkSources_946851_54.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cf7568ed-8702-4f3c-b0bc-b598ec703712_isrForCrosstalkSources_946851_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030213 in provider escience_a40 proc_lsst.multi INFO: found job 17030213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030214 in provider escience_a40 proc_lsst.multi INFO: found job 17030214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030215 in provider escience_a40 proc_lsst.multi INFO: found job 17030215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2898246ee41af83127519b28e68ab3bb with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8c8a7d78-063e-4ade-ae86-168ac93dd1fd_isrForCrosstalkSources_946959_10.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8c8a7d78-063e-4ade-ae86-168ac93dd1fd_isrForCrosstalkSources_946959_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030216 in provider escience_a40 proc_lsst.multi INFO: found job 17030216 in provider escience_a40 parsl.dataflow.memoization DEBUG: Storing app cache entry 7e92609887c0e3c3a14b4e46fa1fad10 with result from task 61 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cd796337-65bf-468e-8fe1-9aa730266465_isrForCrosstalkSources_946851_27.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/cd796337-65bf-468e-8fe1-9aa730266465_isrForCrosstalkSources_946851_27.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61f482fc2ea2c1e8b252dec6e6613797 with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/98f66367-8e96-4bcc-9f76-a175e7356a91_isrForCrosstalkSources_946957_7.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/98f66367-8e96-4bcc-9f76-a175e7356a91_isrForCrosstalkSources_946957_7.stderr parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97ebff8c4e88c10630c02a1287dc606f with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ff7bfff9-adb2-4820-9cd7-2ec5b30047c8_isrForCrosstalkSources_946956_34.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ff7bfff9-adb2-4820-9cd7-2ec5b30047c8_isrForCrosstalkSources_946956_34.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 311170dab98dc928a511e9a7e70e9cf7 with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/85a6d340-611c-4aea-8d05-d130c06c899e_isrForCrosstalkSources_946853_37.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/85a6d340-611c-4aea-8d05-d130c06c899e_isrForCrosstalkSources_946853_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:57 proc_lsst.multi:146 [INFO] found job 17030217 in provider escience_a40 proc_lsst.multi INFO: found job 17030217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030218 in provider checkpoint proc_lsst.multi INFO: found job 17030218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5faeda740fb32cb1288e7a0b21c2ec7 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7e522f20-2ae9-40b8-95e9-91b7d4a1730d_isrForCrosstalkSources_946958_2.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7e522f20-2ae9-40b8-95e9-91b7d4a1730d_isrForCrosstalkSources_946958_2.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f036e1a1056993a76954c1e2ca2a6c2 with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/285d4677-9966-49a3-9eb3-47a2e1fd03bf_isrForCrosstalkSources_946852_38.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/285d4677-9966-49a3-9eb3-47a2e1fd03bf_isrForCrosstalkSources_946852_38.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f9bbab548eddad71c7bdde3557f5a1 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eda54860-e98b-4d4a-9aca-8992e458866e_isrForCrosstalkSources_946851_38.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eda54860-e98b-4d4a-9aca-8992e458866e_isrForCrosstalkSources_946851_38.stderr parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a92f81f490ae5ce0ad0a3cf4ade07db9 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3fccc2a1-c7fd-4e79-a588-b442ba0efd19_isrForCrosstalkSources_946959_56.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3fccc2a1-c7fd-4e79-a588-b442ba0efd19_isrForCrosstalkSources_946959_56.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18ea3b5e200a7f8b294f388759b76053 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e07f42e-c418-4d0c-b968-54e3f25ff088_isrForCrosstalkSources_946959_50.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e07f42e-c418-4d0c-b968-54e3f25ff088_isrForCrosstalkSources_946959_50.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4e0d95770e67b015f21d5137747d2a5 with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6a839b9-90db-49a8-ada9-a93b0ae16374_isrForCrosstalkSources_946853_35.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6a839b9-90db-49a8-ada9-a93b0ae16374_isrForCrosstalkSources_946853_35.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce785687e0416474bec2e369037bd0d1 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/613e8338-3853-4140-a434-de0f217c7ee3_isrForCrosstalkSources_946958_6.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/613e8338-3853-4140-a434-de0f217c7ee3_isrForCrosstalkSources_946958_6.stderr parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55158fce0b39bcb883af77a1edee026c with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a965af7b-b2a1-4fc1-871b-40d5fe13dac9_isrForCrosstalkSources_946853_20.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a965af7b-b2a1-4fc1-871b-40d5fe13dac9_isrForCrosstalkSources_946853_20.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532eeaf32f5ee6d3347c45c63e3582df with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a187315b-6263-4949-9120-8d2fa6927687_isrForCrosstalkSources_946959_2.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a187315b-6263-4949-9120-8d2fa6927687_isrForCrosstalkSources_946959_2.stderr parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea723e406046fd3e3fcfbe9c0f4cdf4d with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cdd65df2-d425-4f5f-a375-760bc997cdc3_isrForCrosstalkSources_946853_54.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cdd65df2-d425-4f5f-a375-760bc997cdc3_isrForCrosstalkSources_946853_54.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5f15e8bd742c2ee3d0eda74b0d161eb with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a3b2445-e54a-48a5-ae4b-f2aadf4c83ba_isrForCrosstalkSources_946957_33.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a3b2445-e54a-48a5-ae4b-f2aadf4c83ba_isrForCrosstalkSources_946957_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030219 in provider checkpoint proc_lsst.multi INFO: found job 17030219 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030220 in provider checkpoint proc_lsst.multi INFO: found job 17030220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29003628b1e6ae471e6491adb3b59213 with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a8b091c6-b231-4570-b43f-4cae46dec519_isrForCrosstalkSources_946956_61.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a8b091c6-b231-4570-b43f-4cae46dec519_isrForCrosstalkSources_946956_61.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e61221de9ed19d1add6454368334ba with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/ebad2e87-fcf7-4619-b608-67fd2f243fc5_isrForCrosstalkSources_946959_37.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/ebad2e87-fcf7-4619-b608-67fd2f243fc5_isrForCrosstalkSources_946959_37.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 204f5c01b1c710c118af18157a705527 with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/635de5af-169e-41df-8275-197253a3c1b9_isrForCrosstalkSources_946958_20.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/635de5af-169e-41df-8275-197253a3c1b9_isrForCrosstalkSources_946958_20.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7df0945c16f7af4349f864bd69b90e1f with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b55da2fa-dc86-42da-9961-94c443c501da_isrForCrosstalkSources_946956_6.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b55da2fa-dc86-42da-9961-94c443c501da_isrForCrosstalkSources_946956_6.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf41e08a4c26db6056509b96eb11e1b with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7ef3f91b-a5f3-45da-8a54-52f8b73281cf_isrForCrosstalkSources_946852_45.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7ef3f91b-a5f3-45da-8a54-52f8b73281cf_isrForCrosstalkSources_946852_45.stderr parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry adefd47ba98739e2c96b53ec9f530819 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f712c01a-69f5-4f4f-bc51-70d7307c7614_isrForCrosstalkSources_946853_11.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f712c01a-69f5-4f4f-bc51-70d7307c7614_isrForCrosstalkSources_946853_11.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80739ce6a80809f28825479a7c21a52a with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/64dd05f0-0cec-4272-b781-4dbf9fc1b531_isrForCrosstalkSources_946956_41.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/64dd05f0-0cec-4272-b781-4dbf9fc1b531_isrForCrosstalkSources_946956_41.stderr parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4e7de03c017ebf62d6e2838276cfc92 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b63a9bee-c3ec-410c-b797-27a76ba5584f_isrForCrosstalkSources_946956_21.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b63a9bee-c3ec-410c-b797-27a76ba5584f_isrForCrosstalkSources_946956_21.stderr parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7e2b6f4cfaeb0b9fb96440e5950dafb with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/994c2669-a63e-498f-90f4-5701d88c6c3e_isrForCrosstalkSources_946958_41.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/994c2669-a63e-498f-90f4-5701d88c6c3e_isrForCrosstalkSources_946958_41.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97a6723fe4d187e65f142b4457f6dd46 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/24eee0b2-6e20-4b5c-8def-738f26636dcd_isrForCrosstalkSources_946851_3.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/24eee0b2-6e20-4b5c-8def-738f26636dcd_isrForCrosstalkSources_946851_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030221 in provider checkpoint proc_lsst.multi INFO: found job 17030221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e62fdb2d6ef3fae01cbef669e8cce030 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f2ec0e8f-f58f-4c8f-9e30-25474e805c68_isrForCrosstalkSources_946852_35.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f2ec0e8f-f58f-4c8f-9e30-25474e805c68_isrForCrosstalkSources_946852_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030222 in provider checkpoint proc_lsst.multi INFO: found job 17030222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030223 in provider checkpoint proc_lsst.multi INFO: found job 17030223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030224 in provider checkpoint proc_lsst.multi INFO: found job 17030224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:58 proc_lsst.multi:146 [INFO] found job 17030225 in provider checkpoint proc_lsst.multi INFO: found job 17030225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030226 in provider checkpoint proc_lsst.multi INFO: found job 17030226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030227 in provider checkpoint proc_lsst.multi INFO: found job 17030227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030228 in provider checkpoint proc_lsst.multi INFO: found job 17030228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030229 in provider checkpoint proc_lsst.multi INFO: found job 17030229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030230 in provider checkpoint proc_lsst.multi INFO: found job 17030230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030231 in provider checkpoint proc_lsst.multi INFO: found job 17030231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030232 in provider checkpoint proc_lsst.multi INFO: found job 17030232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:33:59 proc_lsst.multi:146 [INFO] found job 17030233 in provider checkpoint proc_lsst.multi INFO: found job 17030233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030234 in provider checkpoint proc_lsst.multi INFO: found job 17030234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030235 in provider checkpoint proc_lsst.multi INFO: found job 17030235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030236 in provider checkpoint proc_lsst.multi INFO: found job 17030236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030237 in provider checkpoint proc_lsst.multi INFO: found job 17030237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030238 in provider checkpoint proc_lsst.multi INFO: found job 17030238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030239 in provider checkpoint proc_lsst.multi INFO: found job 17030239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030240 in provider checkpoint proc_lsst.multi INFO: found job 17030240 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:00 proc_lsst.multi:146 [INFO] found job 17030241 in provider checkpoint proc_lsst.multi INFO: found job 17030241 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030242 in provider checkpoint proc_lsst.multi INFO: found job 17030242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030243 in provider checkpoint proc_lsst.multi INFO: found job 17030243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 460999aa42140cde789ac31b88622b3e with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/77072f22-6e49-49af-bcc9-8f5c680d72c6_isrForCrosstalkSources_946853_58.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/77072f22-6e49-49af-bcc9-8f5c680d72c6_isrForCrosstalkSources_946853_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 3a3bef5ddcbfafaafbf157942ac77caf with result from task 9 parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030244 in provider checkpoint proc_lsst.multi INFO: found job 17030244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0d104b3e-6b3f-42e9-9991-b2d41ee888f3_isrForCrosstalkSources_946852_7.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0d104b3e-6b3f-42e9-9991-b2d41ee888f3_isrForCrosstalkSources_946852_7.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a676f03b35aacf822bfb34abff806e8 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/09f1a45f-58fe-4673-bb62-72ff1c757c81_isrForCrosstalkSources_946851_30.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/09f1a45f-58fe-4673-bb62-72ff1c757c81_isrForCrosstalkSources_946851_30.stderr parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c815947bcdfbd9a9c18a7e72a897915e with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8a2eb305-086a-4cdd-9d0f-8ada85ec5963_isrForCrosstalkSources_946853_18.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8a2eb305-086a-4cdd-9d0f-8ada85ec5963_isrForCrosstalkSources_946853_18.stderr parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2885cda43120e5393b226109ba01200 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dd7dbf63-246a-4067-ada2-abd447f2e4e2_isrForCrosstalkSources_946959_17.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dd7dbf63-246a-4067-ada2-abd447f2e4e2_isrForCrosstalkSources_946959_17.stderr parsl.dataflow.dflow INFO: Task 8 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85ddc7aa8771c885af305ca733b1ad7 with result from task 8 parsl.dataflow.dflow DEBUG: Task 8 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/dd020585-88b6-4a3c-88c8-a2d1d313de1c_isrForCrosstalkSources_946851_46.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/dd020585-88b6-4a3c-88c8-a2d1d313de1c_isrForCrosstalkSources_946851_46.stderr parsl.dataflow.dflow INFO: Task 16 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f80a46010b68d36c9a7c50b0569360d with result from task 16 parsl.dataflow.dflow DEBUG: Task 16 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/68b21b9e-fb51-4b53-bff6-dec4800f2daa_isrForCrosstalkSources_946957_54.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/68b21b9e-fb51-4b53-bff6-dec4800f2daa_isrForCrosstalkSources_946957_54.stderr parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3b1866a5ca9c25587c23d2d759385f4 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/45bb6713-380e-4fcd-934f-1ec03e05b9c5_isrForCrosstalkSources_946957_13.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/45bb6713-380e-4fcd-934f-1ec03e05b9c5_isrForCrosstalkSources_946957_13.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fccfa3f671d0752ed318de43459398df with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6092ed1-9e73-4c4c-a02d-cda54dac4def_isrForCrosstalkSources_946853_26.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f6092ed1-9e73-4c4c-a02d-cda54dac4def_isrForCrosstalkSources_946853_26.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44fc15cad83bbcdf9086b834ac90adad with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/80a3ec26-1ddb-46f4-9e5e-5afc7c90c0c9_isrForCrosstalkSources_946958_61.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/80a3ec26-1ddb-46f4-9e5e-5afc7c90c0c9_isrForCrosstalkSources_946958_61.stderr parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1c484019c6a4dd27dd12f0b3c654e47 with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/c81d2413-5714-482c-a24d-56f66ffdc6e8_isrForCrosstalkSources_946851_11.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/c81d2413-5714-482c-a24d-56f66ffdc6e8_isrForCrosstalkSources_946851_11.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f949d839a9d154cbc7df432032d3dc3 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b38f83b2-7431-4afe-a52b-4e0a328a9833_isrForCrosstalkSources_946957_12.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b38f83b2-7431-4afe-a52b-4e0a328a9833_isrForCrosstalkSources_946957_12.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5b32ae97d297797de3df75ad645c48f with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/8543ec21-af6d-4963-8532-c4c464895522_isrForCrosstalkSources_946958_16.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/8543ec21-af6d-4963-8532-c4c464895522_isrForCrosstalkSources_946958_16.stderr parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b329cab496213fed71d7a2749b50eae2 with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a59fed00-5920-4c4e-8ecf-89e3159b2587_isrForCrosstalkSources_946956_15.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a59fed00-5920-4c4e-8ecf-89e3159b2587_isrForCrosstalkSources_946956_15.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 321219af2439ba3dbefae6a96816288b with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/521166d5-e7c6-4c39-9548-3d585ef5ce9c_isrForCrosstalkSources_946852_3.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/521166d5-e7c6-4c39-9548-3d585ef5ce9c_isrForCrosstalkSources_946852_3.stderr parsl.dataflow.dflow INFO: Task 17 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 179993a1de436dd2f0cbb43ebe815f7f with result from task 17 parsl.dataflow.dflow DEBUG: Task 17 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a50dc54c-dcce-4af3-a867-04d153de30a1_isrForCrosstalkSources_946958_30.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a50dc54c-dcce-4af3-a867-04d153de30a1_isrForCrosstalkSources_946958_30.stderr parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1432ccb3352a7350fc92b657790b993d with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/81bf828e-f0a7-4b1d-921b-ce1db454d716_isrForCrosstalkSources_946957_24.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/81bf828e-f0a7-4b1d-921b-ce1db454d716_isrForCrosstalkSources_946957_24.stderr parsl.dataflow.dflow INFO: Task 7 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58acf0dfe92472350a4bedbc037fabc9 with result from task 7 parsl.dataflow.dflow DEBUG: Task 7 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/85f122c3-68c5-40e2-8e06-67488aefd0f5_isrForCrosstalkSources_946851_44.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/85f122c3-68c5-40e2-8e06-67488aefd0f5_isrForCrosstalkSources_946851_44.stderr parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c790ce79f4e56cf845a73750d5c2427e with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0483cb62-d1f6-46eb-a078-7f4f68204922_isrForCrosstalkSources_946852_44.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/0483cb62-d1f6-46eb-a078-7f4f68204922_isrForCrosstalkSources_946852_44.stderr parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86da97bca341a3dde7a432f6e0e3a120 with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7171f884-e3f9-477c-8c12-d230470d1447_isrForCrosstalkSources_946853_30.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/7171f884-e3f9-477c-8c12-d230470d1447_isrForCrosstalkSources_946853_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030245 in provider checkpoint proc_lsst.multi INFO: found job 17030245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030246 in provider checkpoint proc_lsst.multi INFO: found job 17030246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030247 in provider checkpoint proc_lsst.multi INFO: found job 17030247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030248 in provider checkpoint proc_lsst.multi INFO: found job 17030248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030249 in provider checkpoint proc_lsst.multi INFO: found job 17030249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:01 proc_lsst.multi:146 [INFO] found job 17030250 in provider checkpoint proc_lsst.multi INFO: found job 17030250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030251 in provider checkpoint proc_lsst.multi INFO: found job 17030251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030252 in provider checkpoint proc_lsst.multi INFO: found job 17030252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030253 in provider checkpoint proc_lsst.multi INFO: found job 17030253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030254 in provider checkpoint proc_lsst.multi INFO: found job 17030254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba10b4efc5a1e893bfbb4695d39b671e with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fd536db6-f29a-425a-85f8-ed301fe50cc2_isrForCrosstalkSources_946853_36.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fd536db6-f29a-425a-85f8-ed301fe50cc2_isrForCrosstalkSources_946853_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030255 in provider checkpoint proc_lsst.multi INFO: found job 17030255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030257 in provider checkpoint proc_lsst.multi INFO: found job 17030257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030258 in provider checkpoint proc_lsst.multi INFO: found job 17030258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:02 proc_lsst.multi:146 [INFO] found job 17030259 in provider checkpoint proc_lsst.multi INFO: found job 17030259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030260 in provider checkpoint proc_lsst.multi INFO: found job 17030260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030261 in provider checkpoint proc_lsst.multi INFO: found job 17030261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030262 in provider checkpoint proc_lsst.multi INFO: found job 17030262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030263 in provider checkpoint proc_lsst.multi INFO: found job 17030263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030264 in provider checkpoint proc_lsst.multi INFO: found job 17030264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030265 in provider checkpoint proc_lsst.multi INFO: found job 17030265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030266 in provider checkpoint proc_lsst.multi INFO: found job 17030266 in provider checkpoint parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.memoization DEBUG: Storing app cache entry 90af4553b76ea2935d4069570752adab with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a3654626-6b64-49d1-a025-1c6f260f2abc_isrForCrosstalkSources_946956_39.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a3654626-6b64-49d1-a025-1c6f260f2abc_isrForCrosstalkSources_946956_39.stderr parsl.dataflow.dflow INFO: Task 83 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cf8b7431b5381009cf834dffc299cce with result from task 83 parsl.dataflow.dflow DEBUG: Task 83 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2695ffa7-ffce-41f3-8aaa-27c7802bd1cb_isrForCrosstalkSources_946959_48.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2695ffa7-ffce-41f3-8aaa-27c7802bd1cb_isrForCrosstalkSources_946959_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:03 proc_lsst.multi:146 [INFO] found job 17030267 in provider checkpoint proc_lsst.multi INFO: found job 17030267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 82 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 564b24c43252fc0d126134b67a26c836 with result from task 82 parsl.dataflow.dflow DEBUG: Task 82 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/672db64a-67a6-4f4d-87b6-60f02e9bd6e9_isrForCrosstalkSources_946957_56.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/672db64a-67a6-4f4d-87b6-60f02e9bd6e9_isrForCrosstalkSources_946957_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:04 proc_lsst.multi:146 [INFO] found job 17030268 in provider checkpoint proc_lsst.multi INFO: found job 17030268 in provider checkpoint parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.memoization DEBUG: Storing app cache entry 5ec2a25c20e3e1537683d02e0d5a690d with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6e97c6ac-e8ef-4fda-ac47-7d47f162426c_isrForCrosstalkSources_946956_13.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6e97c6ac-e8ef-4fda-ac47-7d47f162426c_isrForCrosstalkSources_946956_13.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e925f481d1ac80abc9ad0674ed9dcaa with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/dae5997e-2192-453d-92b0-d9ab1f38ef3c_isrForCrosstalkSources_946852_25.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/dae5997e-2192-453d-92b0-d9ab1f38ef3c_isrForCrosstalkSources_946852_25.stderr parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b3072b29b4b39d023a59a2317f1ff3f with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/916c936d-6d26-4e83-818c-79c739323d1b_isrForCrosstalkSources_946958_14.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/916c936d-6d26-4e83-818c-79c739323d1b_isrForCrosstalkSources_946958_14.stderr parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3b6eb3add0b489ca520d9ba6d318ccad with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a105c537-e93b-4654-a7a5-87246daeb6a0_isrForCrosstalkSources_946958_50.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a105c537-e93b-4654-a7a5-87246daeb6a0_isrForCrosstalkSources_946958_50.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e300de4908a8e0b6ab51d30f4cc92582 with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/71c2beff-47cf-426d-997f-67b323b965ce_isrForCrosstalkSources_946958_52.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/71c2beff-47cf-426d-997f-67b323b965ce_isrForCrosstalkSources_946958_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:04 proc_lsst.multi:146 [INFO] found job 17030269 in provider checkpoint proc_lsst.multi INFO: found job 17030269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:04 proc_lsst.multi:146 [INFO] found job 17030270 in provider checkpoint proc_lsst.multi INFO: found job 17030270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:04 proc_lsst.multi:146 [INFO] found job 17030271 in provider checkpoint proc_lsst.multi INFO: found job 17030271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 346 parsl.dataflow.strategy DEBUG: Executor multi has 346 active tasks, 109/0 running/pending blocks, and 441 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 346 parsl.dataflow.strategy DEBUG: Executor multi has 346 active tasks, 109/0 running/pending blocks, and 441 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6038bee6c2a843be8fa0ca91f23d7250 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6a67ec7-54a2-4a65-a0e5-63894a27c698_isrForCrosstalkSources_946851_32.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6a67ec7-54a2-4a65-a0e5-63894a27c698_isrForCrosstalkSources_946851_32.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36963add90de15ea5a8038d45a1ffca6 with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3b6e69e8-ba92-42ec-8b58-c1f427717275_isrForCrosstalkSources_946851_51.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3b6e69e8-ba92-42ec-8b58-c1f427717275_isrForCrosstalkSources_946851_51.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9db3ef1e71d3ecfac3e738b588a09058 with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b4883db6-80a9-4396-a7e6-f2adc3a3c520_isrForCrosstalkSources_946956_47.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b4883db6-80a9-4396-a7e6-f2adc3a3c520_isrForCrosstalkSources_946956_47.stderr parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3957f4fadf916a8e32d6bb975aed5265 with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bcf89db7-8b1e-4594-890b-229db6638d23_isrForCrosstalkSources_946852_22.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bcf89db7-8b1e-4594-890b-229db6638d23_isrForCrosstalkSources_946852_22.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d97896dc718b9877f8154ec3776914e with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b993e6b1-f7c8-42d3-83a7-7be1ff5a9330_isrForCrosstalkSources_946853_32.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b993e6b1-f7c8-42d3-83a7-7be1ff5a9330_isrForCrosstalkSources_946853_32.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d95aef5caf43e70f7b540356b012cca with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/83371bf9-aca8-4258-ba6d-1ea9e73676f3_isrForCrosstalkSources_946853_51.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/83371bf9-aca8-4258-ba6d-1ea9e73676f3_isrForCrosstalkSources_946853_51.stderr parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 339f2063ba057d3c5938085c744bc03a with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae3dc49a-e294-4ab3-9048-50595a4e1c06_isrForCrosstalkSources_946957_34.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae3dc49a-e294-4ab3-9048-50595a4e1c06_isrForCrosstalkSources_946957_34.stderr parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db0337dbae049ce49c645af7dbd6c65e with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9b034887-c89f-474f-a2b6-48f0f6336f38_isrForCrosstalkSources_946959_26.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9b034887-c89f-474f-a2b6-48f0f6336f38_isrForCrosstalkSources_946959_26.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d6985c63655d6fa510ab602c2214613 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7badad71-61cf-4d5d-9c8f-1a8847c756c9_isrForCrosstalkSources_946851_26.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7badad71-61cf-4d5d-9c8f-1a8847c756c9_isrForCrosstalkSources_946851_26.stderr parsl.dataflow.dflow INFO: Task 107 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ede8e22165d009f8d527110bc3eedbcc with result from task 107 parsl.dataflow.dflow DEBUG: Task 107 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/476ac725-3964-4827-a3f0-888db1b9508c_isrForCrosstalkSources_946958_42.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/476ac725-3964-4827-a3f0-888db1b9508c_isrForCrosstalkSources_946958_42.stderr parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63be1fb67ba0a1c0913ee8355177985a with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/135694f9-4bda-463f-9c45-5e9ae1f63fd1_isrForCrosstalkSources_946959_24.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/135694f9-4bda-463f-9c45-5e9ae1f63fd1_isrForCrosstalkSources_946959_24.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28c780347c693eac14e36e2bcce4fb6c with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/1bd0a8fc-5d70-4fc0-91f4-3c44d8056bc2_isrForCrosstalkSources_946956_50.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/1bd0a8fc-5d70-4fc0-91f4-3c44d8056bc2_isrForCrosstalkSources_946956_50.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 395bb772cb17fee84fc394f115d755ab with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9c24173a-7b85-4953-8677-321de55e37a1_isrForCrosstalkSources_946959_6.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9c24173a-7b85-4953-8677-321de55e37a1_isrForCrosstalkSources_946959_6.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91930f9e79b907231e968b04c300421b with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9c7e3ed8-0580-45d6-aa2b-ba2629653383_isrForCrosstalkSources_946956_24.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9c7e3ed8-0580-45d6-aa2b-ba2629653383_isrForCrosstalkSources_946956_24.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d9f2c804736fca85ef0437424db4f5e with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/187f904d-7e96-4f7e-bce6-6602426bce6d_isrForCrosstalkSources_946957_47.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/187f904d-7e96-4f7e-bce6-6602426bce6d_isrForCrosstalkSources_946957_47.stderr parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 521aff506df6e243086b1ec6388352f5 with result from task 105 parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/702d740b-173f-413f-b419-b4c314fec7be_isrForCrosstalkSources_946852_42.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/702d740b-173f-413f-b419-b4c314fec7be_isrForCrosstalkSources_946852_42.stderr parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d59cbfd7ebe5a439cc49a0433925d4b with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/64f063dd-df79-4870-96b2-9e3f50e456be_isrForCrosstalkSources_946853_43.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/64f063dd-df79-4870-96b2-9e3f50e456be_isrForCrosstalkSources_946853_43.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6986eae2f55ff709457655d3fa14a4d0 with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e65cbbab-ea2d-4d17-a2e8-28850d03396d_isrForCrosstalkSources_946958_53.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e65cbbab-ea2d-4d17-a2e8-28850d03396d_isrForCrosstalkSources_946958_53.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c67ade165f000698a2a4cc2082f3a89b with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4564e026-61aa-4b6f-80ea-3fdcde125ece_isrForCrosstalkSources_946958_24.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4564e026-61aa-4b6f-80ea-3fdcde125ece_isrForCrosstalkSources_946958_24.stderr parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9f406c8c0e4efd9c2e8e46021086b6d with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a5672902-9d25-4126-be58-5fe8ab9c68fc_isrForCrosstalkSources_946958_51.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/a5672902-9d25-4126-be58-5fe8ab9c68fc_isrForCrosstalkSources_946958_51.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f32678f5bce6186215f76f7ad77e64 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd367e53-a91f-4509-b709-6400e20399d3_isrForCrosstalkSources_946852_28.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd367e53-a91f-4509-b709-6400e20399d3_isrForCrosstalkSources_946852_28.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec1e51ef487888a0459bb910127034da with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9fc2e32c-f3a7-494e-81c3-d779403e3f48_isrForCrosstalkSources_946851_36.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9fc2e32c-f3a7-494e-81c3-d779403e3f48_isrForCrosstalkSources_946851_36.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddf8797cd22bf8fc5e2e120791631397 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7f5fa25-2e7b-4122-819e-7972629f00b3_isrForCrosstalkSources_946852_2.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7f5fa25-2e7b-4122-819e-7972629f00b3_isrForCrosstalkSources_946852_2.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 884eae9e3680d079897b0732b0e396bf with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/31bf661b-cf29-4487-98b2-2e12b6afed99_isrForCrosstalkSources_946851_49.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/31bf661b-cf29-4487-98b2-2e12b6afed99_isrForCrosstalkSources_946851_49.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b12f344e68aa73acf4c56ee8ddc8cfb2 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae48b2a1-4cf3-4526-b267-c6aca056ac56_isrForCrosstalkSources_946957_52.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ae48b2a1-4cf3-4526-b267-c6aca056ac56_isrForCrosstalkSources_946957_52.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 856fc65f108b155b3a6ed61587047ad8 with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/11b7bb77-ab60-46a4-9d9f-21be4155b093_isrForCrosstalkSources_946957_45.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/11b7bb77-ab60-46a4-9d9f-21be4155b093_isrForCrosstalkSources_946957_45.stderr parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 145ac4a8b203cac93ce37ef39c5074fa with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/90c27d9d-a5fe-4502-ab5c-20fca9889582_isrForCrosstalkSources_946959_51.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/90c27d9d-a5fe-4502-ab5c-20fca9889582_isrForCrosstalkSources_946959_51.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa68ea0dd8916c72d75325713e8d226e with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0a9d6551-f4b8-489b-be1b-29798d2918eb_isrForCrosstalkSources_946957_58.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/0a9d6551-f4b8-489b-be1b-29798d2918eb_isrForCrosstalkSources_946957_58.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bda8d2d7836125be8a50a716b3efcf8d with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f37b2b9a-c483-484e-88da-2c5dda365b13_isrForCrosstalkSources_946958_59.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f37b2b9a-c483-484e-88da-2c5dda365b13_isrForCrosstalkSources_946958_59.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83d005edd200b4005120a985f9ec1966 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8e1673ad-f793-4164-8c79-ebaccc5e9a85_isrForCrosstalkSources_946959_52.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8e1673ad-f793-4164-8c79-ebaccc5e9a85_isrForCrosstalkSources_946959_52.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 316 parsl.dataflow.strategy DEBUG: Executor multi has 316 active tasks, 109/0 running/pending blocks, and 441 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab51c3e229decd7a1eccbe4bc3bd2a45 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/01f684cd-862a-4303-a9ad-9d70b9f252b7_isrForCrosstalkSources_946851_48.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/01f684cd-862a-4303-a9ad-9d70b9f252b7_isrForCrosstalkSources_946851_48.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 732ff87581b24c6d7fb5a7f723ed1e88 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/aecea867-94d0-436c-9a30-f582ef13ef23_isrForCrosstalkSources_946851_62.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/aecea867-94d0-436c-9a30-f582ef13ef23_isrForCrosstalkSources_946851_62.stderr parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 860c0332b2600e4fddfaac702e46ffae with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f4101a39-75c8-413d-a9dd-80392022988c_isrForCrosstalkSources_946959_36.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f4101a39-75c8-413d-a9dd-80392022988c_isrForCrosstalkSources_946959_36.stderr parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 511ebebf1d2a4c9166eaf9f176cef0fc with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e25030d-a631-4ef9-8652-977b04cf6af8_isrForCrosstalkSources_946959_3.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e25030d-a631-4ef9-8652-977b04cf6af8_isrForCrosstalkSources_946959_3.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c093ba70b3f9b70e0ef8487df046879 with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/de8217ed-5025-449c-942d-e5b889cd05cd_isrForCrosstalkSources_946853_60.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/de8217ed-5025-449c-942d-e5b889cd05cd_isrForCrosstalkSources_946853_60.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfa9b08b78a2ab551c0220826af36568 with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b7c010fb-e94b-4475-b378-750f8d91bf0c_isrForCrosstalkSources_946851_24.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b7c010fb-e94b-4475-b378-750f8d91bf0c_isrForCrosstalkSources_946851_24.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8930bbb63f34a0f053c96f30c4be685b with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/4c65543a-7dea-41a3-8591-44781f345d96_isrForCrosstalkSources_946957_55.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/4c65543a-7dea-41a3-8591-44781f345d96_isrForCrosstalkSources_946957_55.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba48bd84a5c983110a0be6fdf5cd206c with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b5f09db4-00a9-4ed8-a245-dd9361406c04_isrForCrosstalkSources_946959_28.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b5f09db4-00a9-4ed8-a245-dd9361406c04_isrForCrosstalkSources_946959_28.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e5f5cad5452d8a225136d5c3295a13 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f79ff47c-412e-49be-ac4f-c7c3e41c2ce1_isrForCrosstalkSources_946957_14.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f79ff47c-412e-49be-ac4f-c7c3e41c2ce1_isrForCrosstalkSources_946957_14.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f09d86bb1248d65703d6045f50a543f with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f68e2767-6666-49a2-9ca7-578c8ce75d9c_isrForCrosstalkSources_946957_32.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/f68e2767-6666-49a2-9ca7-578c8ce75d9c_isrForCrosstalkSources_946957_32.stderr parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49901f6a6e6a9cd302ee622f974ce661 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b1149b3-3055-4b0e-97e4-114d28bf4a6b_isrForCrosstalkSources_946851_34.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/6b1149b3-3055-4b0e-97e4-114d28bf4a6b_isrForCrosstalkSources_946851_34.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 40686e602d07cd04cfaa5acf8002d6ad with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb801927-f1df-4536-94f1-1014f360d118_isrForCrosstalkSources_946957_29.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb801927-f1df-4536-94f1-1014f360d118_isrForCrosstalkSources_946957_29.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec76d0410753df960f6a034fc3dd975c with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae003319-37cb-459a-be5c-f2d4681150b9_isrForCrosstalkSources_946853_44.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae003319-37cb-459a-be5c-f2d4681150b9_isrForCrosstalkSources_946853_44.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 429090b164f4adefca671d3359ad5938 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8b8ea9be-c16c-4497-a460-2a670263954b_isrForCrosstalkSources_946957_39.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8b8ea9be-c16c-4497-a460-2a670263954b_isrForCrosstalkSources_946957_39.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c10ef2d4b0d103ee884837567ea738a with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/30dd2597-deae-4013-a0d6-c5f0b781a17a_isrForCrosstalkSources_946959_31.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/30dd2597-deae-4013-a0d6-c5f0b781a17a_isrForCrosstalkSources_946959_31.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04dbdbbded42e768c2bb0fa929b3e964 with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0b803735-622f-4b09-afe5-83078a636266_isrForCrosstalkSources_946853_15.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0b803735-622f-4b09-afe5-83078a636266_isrForCrosstalkSources_946853_15.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d8be62841656f2ed6b30162428b1446 with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7df9e71e-523a-443c-bdb4-973e2307e013_isrForCrosstalkSources_946956_49.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7df9e71e-523a-443c-bdb4-973e2307e013_isrForCrosstalkSources_946956_49.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d70aa2ec807027daff985b0e1947d77 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/d4e46b1f-fe15-4122-9260-85f3273f796b_isrForCrosstalkSources_946853_59.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/d4e46b1f-fe15-4122-9260-85f3273f796b_isrForCrosstalkSources_946853_59.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2f4438a09e4b94f2d9dd7b5633ab0d0 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/d0b090ce-4698-4919-9904-567bfb6b59d5_isrForCrosstalkSources_946852_47.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/d0b090ce-4698-4919-9904-567bfb6b59d5_isrForCrosstalkSources_946852_47.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05e54bb56ec05b0330a85945a9b607c0 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b5d7601a-bb0c-426e-a862-349b5dbd2db2_isrForCrosstalkSources_946956_7.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b5d7601a-bb0c-426e-a862-349b5dbd2db2_isrForCrosstalkSources_946956_7.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 112c9399a0d1371df93289dc9d36e06a with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e9d1c7fc-c7e8-4d4e-b86c-838bf77f0e81_isrForCrosstalkSources_946959_42.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/e9d1c7fc-c7e8-4d4e-b86c-838bf77f0e81_isrForCrosstalkSources_946959_42.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f49d241bc21b16f223039a206bf28e with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/ac2e4586-c1be-47d2-8f3e-dc0252e80282_isrForCrosstalkSources_946851_25.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/ac2e4586-c1be-47d2-8f3e-dc0252e80282_isrForCrosstalkSources_946851_25.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dedfd1bf95e9e10ee0b91fbbb8ebf799 with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/af5741c6-68ae-424b-9df6-93d5415296c0_isrForCrosstalkSources_946957_17.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/af5741c6-68ae-424b-9df6-93d5415296c0_isrForCrosstalkSources_946957_17.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e589ee6119d1c6783ee4fd671fa99798 with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c8802d80-4f2e-4d92-8806-d2f4c3ed5481_isrForCrosstalkSources_946956_18.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c8802d80-4f2e-4d92-8806-d2f4c3ed5481_isrForCrosstalkSources_946956_18.stderr parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c7ced307c60e84d5b046f53d779bb4e with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/dad996a0-de32-4bc2-ae17-70ebffcc5b01_isrForCrosstalkSources_946853_49.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/dad996a0-de32-4bc2-ae17-70ebffcc5b01_isrForCrosstalkSources_946853_49.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8119b7198dc451e38bb37919448e473d with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2ccec44f-ee83-45cb-b543-a06b11acdbad_isrForCrosstalkSources_946959_19.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/2ccec44f-ee83-45cb-b543-a06b11acdbad_isrForCrosstalkSources_946959_19.stderr parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5089023926721bedf2cf33286369273f with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d7c3bf25-b97a-42de-b2df-a9c4e00d129a_isrForCrosstalkSources_946959_8.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d7c3bf25-b97a-42de-b2df-a9c4e00d129a_isrForCrosstalkSources_946959_8.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e0ba0554aaa7508c26ef6f597131262 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fcbe0502-4be8-41fc-a87f-5e81435d3998_isrForCrosstalkSources_946958_17.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fcbe0502-4be8-41fc-a87f-5e81435d3998_isrForCrosstalkSources_946958_17.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4b83ad90955c1645c0a2830e877301d with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9b16ef0a-821d-4cac-8075-4287dfa23c38_isrForCrosstalkSources_946851_8.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9b16ef0a-821d-4cac-8075-4287dfa23c38_isrForCrosstalkSources_946851_8.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1882eb363abae7f7a90b70645192ec3 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e109936b-a9a1-4df1-befa-2bbcf5c621ee_isrForCrosstalkSources_946958_15.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e109936b-a9a1-4df1-befa-2bbcf5c621ee_isrForCrosstalkSources_946958_15.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f04719832ced06efd7af360ebf4a39db with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/6e223a4d-2ac5-484b-86ad-ae9d139b4f48_isrForCrosstalkSources_946958_22.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/6e223a4d-2ac5-484b-86ad-ae9d139b4f48_isrForCrosstalkSources_946958_22.stderr parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9e02018dd4264fb162e9707dc28f2830 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/36809492-884f-477c-8732-8e7674e7ed87_isrForCrosstalkSources_946853_29.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/36809492-884f-477c-8732-8e7674e7ed87_isrForCrosstalkSources_946853_29.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e9aa8419461c3fc3d24ac94593696ac with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b20e565a-7c45-4461-b4c2-f9a11c56d672_isrForCrosstalkSources_946852_54.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b20e565a-7c45-4461-b4c2-f9a11c56d672_isrForCrosstalkSources_946852_54.stderr parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2e863e20ea7de4041b50ba07cc2b491 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4c7ecc56-7c03-43c8-ab7c-20206534c357_isrForCrosstalkSources_946852_33.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4c7ecc56-7c03-43c8-ab7c-20206534c357_isrForCrosstalkSources_946852_33.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7098983b038eee5ce13ccdb72325d860 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/598ea790-0979-4939-93be-de95890287d4_isrForCrosstalkSources_946956_12.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/598ea790-0979-4939-93be-de95890287d4_isrForCrosstalkSources_946956_12.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1aa0b8b88ca62528dc53bbdb1cd78598 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/fadc270f-f639-473b-b796-37ae677aef6f_isrForCrosstalkSources_946959_16.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/fadc270f-f639-473b-b796-37ae677aef6f_isrForCrosstalkSources_946959_16.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72ffd1b08feb0d93dc5704017744a625 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8a2a55bb-a422-4872-b7bf-83d59927c5ea_isrForCrosstalkSources_946959_60.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/8a2a55bb-a422-4872-b7bf-83d59927c5ea_isrForCrosstalkSources_946959_60.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f8e9aa0c629d25eb383ddce41a4d099 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2fdaec1f-7c4a-404e-bd5d-f720ee811f0b_isrForCrosstalkSources_946851_21.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/2fdaec1f-7c4a-404e-bd5d-f720ee811f0b_isrForCrosstalkSources_946851_21.stderr parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 00f6d5f19ff1da5ed867533fcd7e6393 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/457b7c31-f36e-47e2-bf45-215d6aa0f278_isrForCrosstalkSources_946956_51.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/457b7c31-f36e-47e2-bf45-215d6aa0f278_isrForCrosstalkSources_946956_51.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70cb55c8dbc0a6adba147ec4048295bc with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/24e09187-4748-4ab5-a5b0-88617fa70be9_isrForCrosstalkSources_946959_55.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/24e09187-4748-4ab5-a5b0-88617fa70be9_isrForCrosstalkSources_946959_55.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea7c005426a39e9d23d1ebef2cc3f376 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3c09b9a4-4834-4fe2-badd-65d998bdea4b_isrForCrosstalkSources_946958_5.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3c09b9a4-4834-4fe2-badd-65d998bdea4b_isrForCrosstalkSources_946958_5.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5ecd42369add47ac1a37eede3340602 with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a3508f6d-174a-4c0c-9307-f32c22a13917_isrForCrosstalkSources_946853_2.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a3508f6d-174a-4c0c-9307-f32c22a13917_isrForCrosstalkSources_946853_2.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b056d9c5c5346bfedfbd8de2b985b5b8 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/16276c1b-af19-4b23-8e4a-74849e5d49a9_isrForCrosstalkSources_946851_5.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/16276c1b-af19-4b23-8e4a-74849e5d49a9_isrForCrosstalkSources_946851_5.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ad27030e2078303226674e8cc5e56e4 with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c4550e1-4b6f-40ea-b9b3-a4e95a248aa1_isrForCrosstalkSources_946853_23.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c4550e1-4b6f-40ea-b9b3-a4e95a248aa1_isrForCrosstalkSources_946853_23.stderr parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f3866b13de8b025c14fb41841b8fc3c1 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/e48ea48f-40cd-4314-9ae8-dd0b5094615b_isrForCrosstalkSources_946853_34.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/e48ea48f-40cd-4314-9ae8-dd0b5094615b_isrForCrosstalkSources_946853_34.stderr parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6adef19ed8d818f05898f458ba63f0fd with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/f88ad868-5605-4648-8efc-c492356ce349_isrForCrosstalkSources_946851_55.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/f88ad868-5605-4648-8efc-c492356ce349_isrForCrosstalkSources_946851_55.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ee1f1dec8668351cd4183e5d5891677 with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8cd666d8-b6f3-4644-bc8d-6a875bbaeef1_isrForCrosstalkSources_946851_31.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8cd666d8-b6f3-4644-bc8d-6a875bbaeef1_isrForCrosstalkSources_946851_31.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f25dfdd88635de9139a3a0cf4ab166a7 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/31300713-97e5-4ac7-bf49-4a06f5decbcd_isrForCrosstalkSources_946959_46.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/31300713-97e5-4ac7-bf49-4a06f5decbcd_isrForCrosstalkSources_946959_46.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4a9ff4ec130fedaadee12599cefabe2 with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c3ae077-aeed-497b-a3cb-0118e9e34e35_isrForCrosstalkSources_946853_7.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3c3ae077-aeed-497b-a3cb-0118e9e34e35_isrForCrosstalkSources_946853_7.stderr parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4eba74e232064b5e5b9579f924b3378f with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7e5f8933-33b1-4130-804a-d17720579f95_isrForCrosstalkSources_946852_58.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/7e5f8933-33b1-4130-804a-d17720579f95_isrForCrosstalkSources_946852_58.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 010fe0bf2b945ad8ba4d27f1c8801c03 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/130133ae-37f2-4f3f-b13e-0eeb557395c1_isrForCrosstalkSources_946852_26.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/130133ae-37f2-4f3f-b13e-0eeb557395c1_isrForCrosstalkSources_946852_26.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2a8b3c2d94971a0b04df741dd1e2bebf with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/feba39e2-317a-49dc-aed2-138143a4e8a2_isrForCrosstalkSources_946853_12.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/feba39e2-317a-49dc-aed2-138143a4e8a2_isrForCrosstalkSources_946853_12.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edb551bc25c3212c60143838bccb1fbb with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bbaf50a1-424f-431e-9b3f-604eb6f60e91_isrForCrosstalkSources_946957_11.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bbaf50a1-424f-431e-9b3f-604eb6f60e91_isrForCrosstalkSources_946957_11.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc4977a9b66e03f751a489465af166fd with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/12fd4296-8b05-4d22-b463-4c4c95caaea1_isrForCrosstalkSources_946956_17.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/12fd4296-8b05-4d22-b463-4c4c95caaea1_isrForCrosstalkSources_946956_17.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f2aafc404c4cd010577d1c28b306a3b with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ac705e18-1bff-4da0-ada7-67feab9cc26f_isrForCrosstalkSources_946853_13.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ac705e18-1bff-4da0-ada7-67feab9cc26f_isrForCrosstalkSources_946853_13.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 694e8e6125ab4f98cc5edf64978410fe with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1da9433a-53db-44b2-8c1a-16433172ee0b_isrForCrosstalkSources_946958_13.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1da9433a-53db-44b2-8c1a-16433172ee0b_isrForCrosstalkSources_946958_13.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc18bd13b086b929e55e90666a23ca60 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e854cb07-b972-4b9b-847e-8ff295edd2be_isrForCrosstalkSources_946958_57.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e854cb07-b972-4b9b-847e-8ff295edd2be_isrForCrosstalkSources_946958_57.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9687de6090ff87d1e2f0fb23b40e7eaf with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2b95b3a7-210a-4517-be28-a32c3470bae7_isrForCrosstalkSources_946956_35.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2b95b3a7-210a-4517-be28-a32c3470bae7_isrForCrosstalkSources_946956_35.stderr parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 896a3b9fbcbf30506edce8184fea16c2 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fb7b9c1a-ecff-4684-8ac7-ad8135043b33_isrForCrosstalkSources_946958_33.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/fb7b9c1a-ecff-4684-8ac7-ad8135043b33_isrForCrosstalkSources_946958_33.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5a7a86860cacfd69bda91d68e9db966 with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/0591d93e-aeb3-40f2-8b94-26baba3d3420_isrForCrosstalkSources_946959_57.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/0591d93e-aeb3-40f2-8b94-26baba3d3420_isrForCrosstalkSources_946959_57.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295ef31ee16f624c02c4f44b08f427f5 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6da14152-8ca2-4b12-8575-c004d1a95309_isrForCrosstalkSources_946956_54.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/6da14152-8ca2-4b12-8575-c004d1a95309_isrForCrosstalkSources_946956_54.stderr parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1443fcf8d6ae3f4ef04c8375ca5e3a0 with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13bbfc2f-5dc3-40eb-a0e0-6379f56acc55_isrForCrosstalkSources_946959_12.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13bbfc2f-5dc3-40eb-a0e0-6379f56acc55_isrForCrosstalkSources_946959_12.stderr parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89137a9a62c9b2d810171e01cf7886c3 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fb1c2adf-b556-4794-a710-b99d86804ed5_isrForCrosstalkSources_946956_59.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/fb1c2adf-b556-4794-a710-b99d86804ed5_isrForCrosstalkSources_946956_59.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 837d362956af5958b126070662d3e7f4 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/03da1849-c128-474f-ae5f-e15c5831d931_isrForCrosstalkSources_946958_4.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/03da1849-c128-474f-ae5f-e15c5831d931_isrForCrosstalkSources_946958_4.stderr parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 444199f930a34ad80d74ae2d82eb14ef with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/271dc04d-0d9b-4648-8e20-cac03d458562_isrForCrosstalkSources_946852_59.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/271dc04d-0d9b-4648-8e20-cac03d458562_isrForCrosstalkSources_946852_59.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eee8dde7270f118316fb76053eab15d6 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b3a0d573-b2b7-4df7-bbad-ea730418fbcb_isrForCrosstalkSources_946958_55.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b3a0d573-b2b7-4df7-bbad-ea730418fbcb_isrForCrosstalkSources_946958_55.stderr parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 583d790ce7503f924a6a68acee096fa0 with result from task 141 parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/18f3184f-e350-43e1-89b3-842d1a4f5360_isrForCrosstalkSources_946956_60.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/18f3184f-e350-43e1-89b3-842d1a4f5360_isrForCrosstalkSources_946956_60.stderr parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b73fd20d2187aaaff7496c1b5fca54e with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3ee5a670-48e4-485f-b69e-6eea51be6770_isrForCrosstalkSources_946958_48.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/3ee5a670-48e4-485f-b69e-6eea51be6770_isrForCrosstalkSources_946958_48.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62444c7dd772f2423f559a642b78205f with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/33ab279e-42c5-4d7f-a662-257b0f46c673_isrForCrosstalkSources_946852_21.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/33ab279e-42c5-4d7f-a662-257b0f46c673_isrForCrosstalkSources_946852_21.stderr parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68571eb429901b1fcb3ce8e5d1ec3d40 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bfd55b9e-6345-4e51-9330-a8673138dfb8_isrForCrosstalkSources_946851_1.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/bfd55b9e-6345-4e51-9330-a8673138dfb8_isrForCrosstalkSources_946851_1.stderr parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf8be2ded38efcf1885b15703d61f49c with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/db2eb3c8-03ff-41fe-a653-bca21971aa4c_isrForCrosstalkSources_946956_5.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/db2eb3c8-03ff-41fe-a653-bca21971aa4c_isrForCrosstalkSources_946956_5.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fea9ff700e06a204248a7710dd262f28 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/88570778-8bde-47c6-a27f-ac125b2a6419_isrForCrosstalkSources_946957_46.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/88570778-8bde-47c6-a27f-ac125b2a6419_isrForCrosstalkSources_946957_46.stderr parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dcbee32b34e4c80c348daf5ec9dba40 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3ce29ee8-932f-4374-86ff-8e292b20aebd_isrForCrosstalkSources_946851_15.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3ce29ee8-932f-4374-86ff-8e292b20aebd_isrForCrosstalkSources_946851_15.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d317813319245c2297dcf99449aa995 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e3d70eb-583f-491e-86d2-7548fec1e176_isrForCrosstalkSources_946959_40.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1e3d70eb-583f-491e-86d2-7548fec1e176_isrForCrosstalkSources_946959_40.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c8ca4d80f697d0579d37eb1b8d25be4 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/81b8eb58-489c-4e6d-96c9-fbc8fd453bee_isrForCrosstalkSources_946853_4.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/81b8eb58-489c-4e6d-96c9-fbc8fd453bee_isrForCrosstalkSources_946853_4.stderr parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 855d8bd365bcb8f0cdb2a55270545824 with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/abec3ea7-d7b2-44f5-ac9a-c5b6b5bc4f83_isrForCrosstalkSources_946959_44.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/abec3ea7-d7b2-44f5-ac9a-c5b6b5bc4f83_isrForCrosstalkSources_946959_44.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 402b8d12ec47e44f592de5a23b3ad86a with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6b07cb9c-6c80-4a07-9ead-c4e1f76b5f10_isrForCrosstalkSources_946957_37.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6b07cb9c-6c80-4a07-9ead-c4e1f76b5f10_isrForCrosstalkSources_946957_37.stderr parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4508b88791b7a6eb5e357bb88a814933 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1a6a8ee5-cb00-419e-94b6-de06f7b0ca6a_isrForCrosstalkSources_946958_35.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1a6a8ee5-cb00-419e-94b6-de06f7b0ca6a_isrForCrosstalkSources_946958_35.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2a72ab64ec7756ec37cc81efc17612e with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/2cfd5e65-7ec1-4a38-aa2c-eadd5a81f596_isrForCrosstalkSources_946852_8.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/2cfd5e65-7ec1-4a38-aa2c-eadd5a81f596_isrForCrosstalkSources_946852_8.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1c775a53207887606db90a2449a3cd0 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661b811e-f9ce-454b-b230-d4bb4973b510_isrForCrosstalkSources_946956_46.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661b811e-f9ce-454b-b230-d4bb4973b510_isrForCrosstalkSources_946956_46.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d0a8c2cfc20f4e8619852b203a73e21 with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9d227508-7049-494c-a905-5871b60a2005_isrForCrosstalkSources_946851_60.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9d227508-7049-494c-a905-5871b60a2005_isrForCrosstalkSources_946851_60.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7364302e6588f45981e2aaaab185b13 with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dc752a24-d7c5-4960-87ad-a6c3114f2f5d_isrForCrosstalkSources_946959_41.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/dc752a24-d7c5-4960-87ad-a6c3114f2f5d_isrForCrosstalkSources_946959_41.stderr parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5ec630f6e8c7ae10e117ef3785f002c with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/16801739-f6d7-450b-aeb2-b17e9c7b3241_isrForCrosstalkSources_946958_28.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/16801739-f6d7-450b-aeb2-b17e9c7b3241_isrForCrosstalkSources_946958_28.stderr parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcf284138a920fea069bdc14c420c78c with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ef17170a-7897-46de-b231-debbcce5cb60_isrForCrosstalkSources_946957_22.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ef17170a-7897-46de-b231-debbcce5cb60_isrForCrosstalkSources_946957_22.stderr parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96d4a90ea667b637c5cd1c9b06abd8fd with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/30dc6f01-715a-4876-86e1-37a4d1ed4eed_isrForCrosstalkSources_946851_45.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/30dc6f01-715a-4876-86e1-37a4d1ed4eed_isrForCrosstalkSources_946851_45.stderr parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b3f67d210a95f946b1c2718f7fc2ed23 with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fe41a5ae-af21-4246-8f33-3fad2a76b039_isrForCrosstalkSources_946957_2.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fe41a5ae-af21-4246-8f33-3fad2a76b039_isrForCrosstalkSources_946957_2.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17b7e2efa5bf49d7a1014fc55c6f3328 with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a9cb1681-d8c3-43cc-b226-27c62dd4dc01_isrForCrosstalkSources_946853_9.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a9cb1681-d8c3-43cc-b226-27c62dd4dc01_isrForCrosstalkSources_946853_9.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66aeecef34ef7d5c15b2e26411f7170d with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8e1814eb-6d64-4347-85b3-08f3a1b0284d_isrForCrosstalkSources_946956_44.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8e1814eb-6d64-4347-85b3-08f3a1b0284d_isrForCrosstalkSources_946956_44.stderr parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e25c190c5c47daf70553586d5fc82868 with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0eddc91f-a9e6-4453-884f-22381aaf2cee_isrForCrosstalkSources_946853_62.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/0eddc91f-a9e6-4453-884f-22381aaf2cee_isrForCrosstalkSources_946853_62.stderr parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf3bea7952c95c862fe371acfffd4497 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9cd3647a-f39a-4205-b346-7fc64378b713_isrForCrosstalkSources_946851_43.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/9cd3647a-f39a-4205-b346-7fc64378b713_isrForCrosstalkSources_946851_43.stderr parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20288b1d8360d6d5b115dcef4b9f06ef with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9277491c-3630-4a76-a220-e4099fb3cfe8_isrForCrosstalkSources_946956_9.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/9277491c-3630-4a76-a220-e4099fb3cfe8_isrForCrosstalkSources_946956_9.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c263febe7bc60023a66c943668451f77 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/50a13037-0954-4479-9905-20eb0fbb37e7_isrForCrosstalkSources_946959_15.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/50a13037-0954-4479-9905-20eb0fbb37e7_isrForCrosstalkSources_946959_15.stderr parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6e2a8fd6faffe77efa307615d4568e2 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8579151f-20fe-4766-8e8d-ba819709586a_isrForCrosstalkSources_946851_56.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8579151f-20fe-4766-8e8d-ba819709586a_isrForCrosstalkSources_946851_56.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea1b1f1de6ddbda8510d78171e0484d with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1b5b1e0c-d5bc-4f24-9d0d-c09645c29a9c_isrForCrosstalkSources_946957_60.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1b5b1e0c-d5bc-4f24-9d0d-c09645c29a9c_isrForCrosstalkSources_946957_60.stderr parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f28343eeccd77f7c550ff3709258ec96 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a8f33859-b67e-4c6e-8b23-742ffc97e01f_isrForCrosstalkSources_946853_39.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a8f33859-b67e-4c6e-8b23-742ffc97e01f_isrForCrosstalkSources_946853_39.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c9e19b342a9f7173b1f8c488b106bc8 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e095400c-43a0-44d8-bf65-a61c8eb1f4c9_isrForCrosstalkSources_946852_31.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e095400c-43a0-44d8-bf65-a61c8eb1f4c9_isrForCrosstalkSources_946852_31.stderr parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c60e30a23d942e784de3b80b305d97db with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5d870d5a-3e34-464b-bfbc-0b9ae4708957_isrForCrosstalkSources_946957_31.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5d870d5a-3e34-464b-bfbc-0b9ae4708957_isrForCrosstalkSources_946957_31.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9cca8f1ec6b66dcbf1c7967fe2c3df05 with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2b3041ca-bd0f-4855-9054-41a4bc7bc10a_isrForCrosstalkSources_946957_15.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2b3041ca-bd0f-4855-9054-41a4bc7bc10a_isrForCrosstalkSources_946957_15.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 218 parsl.dataflow.strategy DEBUG: Executor multi has 218 active tasks, 109/0 running/pending blocks, and 481 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ca7725a596e3178285046f43535743 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e9c55585-7a05-4264-ae07-23308e2248ad_isrForCrosstalkSources_946851_29.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e9c55585-7a05-4264-ae07-23308e2248ad_isrForCrosstalkSources_946851_29.stderr parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8420e7992e07a68e4826fabce70a1136 with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/91650281-eeec-4ef2-b931-8bea3e2f5c5a_isrForCrosstalkSources_946956_16.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/91650281-eeec-4ef2-b931-8bea3e2f5c5a_isrForCrosstalkSources_946956_16.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a49a232377a96d12158416c353f8ca3b with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a706d2c1-c08b-4d13-8956-f9082c549383_isrForCrosstalkSources_946959_53.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a706d2c1-c08b-4d13-8956-f9082c549383_isrForCrosstalkSources_946959_53.stderr parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36ca2912b1fd71bf811e7b71ade31632 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b8a25931-412c-454a-8ade-6e22125fa1b5_isrForCrosstalkSources_946958_54.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b8a25931-412c-454a-8ade-6e22125fa1b5_isrForCrosstalkSources_946958_54.stderr parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29a1b853aac90bf385c5858722cdf48c with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f2c8a970-73fa-401c-8ed6-91d9f4ec7b2b_isrForCrosstalkSources_946959_30.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f2c8a970-73fa-401c-8ed6-91d9f4ec7b2b_isrForCrosstalkSources_946959_30.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f012c92feb32e9c6e573d0f6d77edda1 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d4ec48f0-b0cd-4b40-ac34-4f46592e59eb_isrForCrosstalkSources_946959_59.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d4ec48f0-b0cd-4b40-ac34-4f46592e59eb_isrForCrosstalkSources_946959_59.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dce2567cfaa7cf739004d800f021e8cb with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c2c8ed97-6651-4583-b776-615ed27b3005_isrForCrosstalkSources_946853_24.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c2c8ed97-6651-4583-b776-615ed27b3005_isrForCrosstalkSources_946853_24.stderr parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7cebc889ab3b321cb0358a07413153a with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2c41d307-1ca3-4960-91bc-15d19f494124_isrForCrosstalkSources_946956_23.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/2c41d307-1ca3-4960-91bc-15d19f494124_isrForCrosstalkSources_946956_23.stderr parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c97f6d5ce0401ac11bf089c5455b2ae with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c43b06f2-0b47-49db-b4cf-4f3590119486_isrForCrosstalkSources_946853_14.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c43b06f2-0b47-49db-b4cf-4f3590119486_isrForCrosstalkSources_946853_14.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d61f822bd037440c49aa9b1e0c8a9c6 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4b8b9054-1596-4c0d-b9ef-b7af5306caf7_isrForCrosstalkSources_946959_43.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4b8b9054-1596-4c0d-b9ef-b7af5306caf7_isrForCrosstalkSources_946959_43.stderr parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bb1b3fb59b1200ffc0ccd8b25853b8b with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a57a457e-c213-4cc9-924f-68e199941d14_isrForCrosstalkSources_946851_59.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a57a457e-c213-4cc9-924f-68e199941d14_isrForCrosstalkSources_946851_59.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c09e3164f8fa7a966d6793fbfd6c462 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c64c4463-f7f6-474e-920d-ba61d989b25c_isrForCrosstalkSources_946853_5.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/c64c4463-f7f6-474e-920d-ba61d989b25c_isrForCrosstalkSources_946853_5.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff28f52341268638c36e74001eba436e with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/31187d00-eb93-4b97-b396-ca9e3881f055_isrForCrosstalkSources_946956_1.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/31187d00-eb93-4b97-b396-ca9e3881f055_isrForCrosstalkSources_946956_1.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51d1c7545b3cafd2fe696913f1cc85b8 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f05154e9-b7d8-4930-a0d1-9876e91bd9dd_isrForCrosstalkSources_946852_20.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f05154e9-b7d8-4930-a0d1-9876e91bd9dd_isrForCrosstalkSources_946852_20.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25de1bc25e8ad0e3654b5553a8508c9d with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6819116-2ac0-44ef-a734-7c6d44dcb3e5_isrForCrosstalkSources_946851_39.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/a6819116-2ac0-44ef-a734-7c6d44dcb3e5_isrForCrosstalkSources_946851_39.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2201a86a712e0f4a3c496fe4ca96162 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2c1eaca0-9060-480e-8f9d-56fc5c8db307_isrForCrosstalkSources_946957_5.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/2c1eaca0-9060-480e-8f9d-56fc5c8db307_isrForCrosstalkSources_946957_5.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98d7120df0ad9371b709c251acb389e2 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7bca1028-fc6b-4fe2-9cb2-3fe76d33c5d2_isrForCrosstalkSources_946851_33.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7bca1028-fc6b-4fe2-9cb2-3fe76d33c5d2_isrForCrosstalkSources_946851_33.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78a54c4722860af36646c108197debf6 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/67539512-d846-4669-b114-7d5a5c35b8ed_isrForCrosstalkSources_946956_53.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/67539512-d846-4669-b114-7d5a5c35b8ed_isrForCrosstalkSources_946956_53.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 873e187c9ab0e62b1509ca9ad95e4e94 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/37c7f323-0ad2-4fc6-9eb6-f0f9dc42a6fc_isrForCrosstalkSources_946853_17.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/37c7f323-0ad2-4fc6-9eb6-f0f9dc42a6fc_isrForCrosstalkSources_946853_17.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25922f8152143cc2b101a432963ddd3a with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/671fdade-0666-48f6-b9bd-5008aaf2d584_isrForCrosstalkSources_946958_27.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/671fdade-0666-48f6-b9bd-5008aaf2d584_isrForCrosstalkSources_946958_27.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 82a673d8dbf2a3ead81cdf43aed4d2fe with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/dda0c2ca-24d3-4c2c-8893-2778b9782baf_isrForCrosstalkSources_946956_14.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/dda0c2ca-24d3-4c2c-8893-2778b9782baf_isrForCrosstalkSources_946956_14.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c981d3ddaadf2a44778144287912844 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/af514fc7-c351-4a3a-b45f-186093de5b25_isrForCrosstalkSources_946956_19.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/af514fc7-c351-4a3a-b45f-186093de5b25_isrForCrosstalkSources_946956_19.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d4f56af4de70919d5deac9a6d580eb2 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7e995e3d-2077-458c-8358-e749bf2501be_isrForCrosstalkSources_946956_27.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7e995e3d-2077-458c-8358-e749bf2501be_isrForCrosstalkSources_946956_27.stderr parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd1b2c0083db5a635dcad0d45819688 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/77a6740c-5fa6-4fa9-b18b-a321d8a628d6_isrForCrosstalkSources_946958_23.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/77a6740c-5fa6-4fa9-b18b-a321d8a628d6_isrForCrosstalkSources_946958_23.stderr parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1e8549dcc94cd74902612374d10640f with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8ae5b373-4042-4235-ac25-a811eb5e847e_isrForCrosstalkSources_946957_40.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8ae5b373-4042-4235-ac25-a811eb5e847e_isrForCrosstalkSources_946957_40.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7d25a61544bd573ea409ae4d82c8af4 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e3373ce6-779c-4a65-b64f-06444327b1bd_isrForCrosstalkSources_946851_14.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/e3373ce6-779c-4a65-b64f-06444327b1bd_isrForCrosstalkSources_946851_14.stderr parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 074949f71141c856a4f455ee631fc46b with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ea11b491-3bf6-455b-960b-9832d0e979fc_isrForCrosstalkSources_946956_32.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ea11b491-3bf6-455b-960b-9832d0e979fc_isrForCrosstalkSources_946956_32.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e72f16f5787e96d1cc282e2300d5e800 with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/43817d7c-aec4-4952-9886-7f0504741277_isrForCrosstalkSources_946958_37.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/43817d7c-aec4-4952-9886-7f0504741277_isrForCrosstalkSources_946958_37.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a10f45c79c718ccf1a361d122ff81dc with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661a8dd2-34ec-4db1-997a-e6458a4218a1_isrForCrosstalkSources_946956_38.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/661a8dd2-34ec-4db1-997a-e6458a4218a1_isrForCrosstalkSources_946956_38.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d060eff6e79546179b9adfbc6a641ec with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/4d9c348f-f881-4292-8634-d9ae96efe863_isrForCrosstalkSources_946956_57.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/4d9c348f-f881-4292-8634-d9ae96efe863_isrForCrosstalkSources_946956_57.stderr parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 548dd7e7e35c7ca8272a8c5119715cb6 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/19cf9db3-16d9-455e-a45a-30cbb534398d_isrForCrosstalkSources_946957_51.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/19cf9db3-16d9-455e-a45a-30cbb534398d_isrForCrosstalkSources_946957_51.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a16b6de9cab0a6a5c584618bcf35d67f with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9553e31b-2442-48f6-b626-b3a72cf10559_isrForCrosstalkSources_946852_11.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9553e31b-2442-48f6-b626-b3a72cf10559_isrForCrosstalkSources_946852_11.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c55605359cbeb984def7a4cd82f51178 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e21e8730-2cb7-4b9c-89a4-528655fef785_isrForCrosstalkSources_946958_62.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/e21e8730-2cb7-4b9c-89a4-528655fef785_isrForCrosstalkSources_946958_62.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4a9baeb1f878595a2c64ae5af0331ab6 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7fefe133-073b-4301-9959-acd94a90ce19_isrForCrosstalkSources_946851_10.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/7fefe133-073b-4301-9959-acd94a90ce19_isrForCrosstalkSources_946851_10.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af94e71433f8c7c1f75b1ed7f2353ae0 with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7749163e-999d-4b04-8a71-175b4e244462_isrForCrosstalkSources_946958_11.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7749163e-999d-4b04-8a71-175b4e244462_isrForCrosstalkSources_946958_11.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e603bc507064ebbdfa168e8eb623f0a1 with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4611f4b2-2455-456b-b02f-395c8546082e_isrForCrosstalkSources_946852_13.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4611f4b2-2455-456b-b02f-395c8546082e_isrForCrosstalkSources_946852_13.stderr parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c7afa0c449201fdeb4e7d4ea2f33991 with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/5b42c2e6-abe1-4a0e-bca5-33ff0e804af5_isrForCrosstalkSources_946852_53.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/5b42c2e6-abe1-4a0e-bca5-33ff0e804af5_isrForCrosstalkSources_946852_53.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2c8c18c10b347875f6a4a461e4acfab with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/2315a55d-3ef7-4b40-8353-ae216e29f436_isrForCrosstalkSources_946958_44.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/2315a55d-3ef7-4b40-8353-ae216e29f436_isrForCrosstalkSources_946958_44.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f3ba8ff6a7c3c6ef122b1de2c32b915 with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/7bc1d307-436b-498b-b7d9-3ce2502bd1c3_isrForCrosstalkSources_946957_28.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/7bc1d307-436b-498b-b7d9-3ce2502bd1c3_isrForCrosstalkSources_946957_28.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2acaf309a2e0563ceceb8d7bfa938b0c with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f6496326-8ccc-4c88-a2cc-d6125194df76_isrForCrosstalkSources_946959_20.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/f6496326-8ccc-4c88-a2cc-d6125194df76_isrForCrosstalkSources_946959_20.stderr parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 64d917fc0966d5ec9a65a1412a1a6c82 with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/387abb9d-54a0-4cca-b482-5dab5a63e17a_isrForCrosstalkSources_946956_20.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/387abb9d-54a0-4cca-b482-5dab5a63e17a_isrForCrosstalkSources_946956_20.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85aa6f54bc1370cb4f1bf7cb671837eb with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/44ea3ff5-7af9-4732-8b1b-7c182903d639_isrForCrosstalkSources_946958_18.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/44ea3ff5-7af9-4732-8b1b-7c182903d639_isrForCrosstalkSources_946958_18.stderr parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9769cd6b8aae75f70527143a42cad4f9 with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f2507248-db43-440d-aeac-7a99225cabde_isrForCrosstalkSources_946958_56.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f2507248-db43-440d-aeac-7a99225cabde_isrForCrosstalkSources_946958_56.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0186e197353d5ac882ad6d327ff4e5d with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bce655a2-794b-4c97-9d2a-4c38ac999f9c_isrForCrosstalkSources_946852_32.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bce655a2-794b-4c97-9d2a-4c38ac999f9c_isrForCrosstalkSources_946852_32.stderr parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cba47e9f4bddff8a1535faac3a375f50 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b99bfbe9-c8a4-4314-ad0f-5e71bd22fd6d_isrForCrosstalkSources_946959_47.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/b99bfbe9-c8a4-4314-ad0f-5e71bd22fd6d_isrForCrosstalkSources_946959_47.stderr parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d45e6d54577267f00bfc23af6467eecc with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/891639aa-d46c-4bff-8d5e-aef729c2b52a_isrForCrosstalkSources_946852_52.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/891639aa-d46c-4bff-8d5e-aef729c2b52a_isrForCrosstalkSources_946852_52.stderr parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6809b73c925424b99ff676247c7898d with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb63fedf-22d1-4d77-ac6e-b0dc4786a101_isrForCrosstalkSources_946851_40.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb63fedf-22d1-4d77-ac6e-b0dc4786a101_isrForCrosstalkSources_946851_40.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22fc0e6906fe1b9efbf31458a60e65c3 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d571452a-0754-406e-a873-17e1624f812e_isrForCrosstalkSources_946956_8.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d571452a-0754-406e-a873-17e1624f812e_isrForCrosstalkSources_946956_8.stderr parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98709d7154344f33094c5db79b855e39 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bf126800-e39f-4ee9-a4d3-d4abf9412d40_isrForCrosstalkSources_946852_39.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bf126800-e39f-4ee9-a4d3-d4abf9412d40_isrForCrosstalkSources_946852_39.stderr parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 72324d367fb09a75fd413f75e0ec7251 with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3b21e03b-c6a1-4a36-9032-7ecbf531db0e_isrForCrosstalkSources_946959_13.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/3b21e03b-c6a1-4a36-9032-7ecbf531db0e_isrForCrosstalkSources_946959_13.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 340177aaf00dfac56f65bb03b2aa44e1 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fff9e9c8-117a-4319-96f4-cc800cb8e3d2_isrForCrosstalkSources_946853_40.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/fff9e9c8-117a-4319-96f4-cc800cb8e3d2_isrForCrosstalkSources_946853_40.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5818978e07b8d2a1f2be1d27cd4b613d with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8349c6e8-bcc7-43dc-825d-f368ec01a5f7_isrForCrosstalkSources_946956_26.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/8349c6e8-bcc7-43dc-825d-f368ec01a5f7_isrForCrosstalkSources_946956_26.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 728e4f19aae692ae4d17ca43100d199b with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8db70aef-3e05-4de1-87f6-960a12e6e5b4_isrForCrosstalkSources_946851_17.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/8db70aef-3e05-4de1-87f6-960a12e6e5b4_isrForCrosstalkSources_946851_17.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d67915996759a5c104e83da11fc3b37 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/481083d8-765b-43ec-acad-a3fe63abcd63_isrForCrosstalkSources_946956_29.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/481083d8-765b-43ec-acad-a3fe63abcd63_isrForCrosstalkSources_946956_29.stderr parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16609e3c57026cb789ff6903bdd91e91 with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6b87c82a-a981-44b8-8053-916033379ef7_isrForCrosstalkSources_946959_4.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6b87c82a-a981-44b8-8053-916033379ef7_isrForCrosstalkSources_946959_4.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38d621ee46c0934fecc0e704493f9062 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7515b70b-788d-49dc-80f5-f54e7be2cdbc_isrForCrosstalkSources_946958_32.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7515b70b-788d-49dc-80f5-f54e7be2cdbc_isrForCrosstalkSources_946958_32.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f33ae314db096df62ba723762dbb11f4 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee0da6cf-8d2c-4f69-af51-3e53f43dc98c_isrForCrosstalkSources_946853_55.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee0da6cf-8d2c-4f69-af51-3e53f43dc98c_isrForCrosstalkSources_946853_55.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19399defe50cbc11188a9145c86d3f98 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/07997d11-d3d3-41cc-9f36-0f5069d1f6ca_isrForCrosstalkSources_946959_61.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/07997d11-d3d3-41cc-9f36-0f5069d1f6ca_isrForCrosstalkSources_946959_61.stderr parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5db94fdbbfffb4a6ecee7b0db5e1f1c with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4b908bb7-8558-48d3-a5c2-94b9c59f5dac_isrForCrosstalkSources_946852_4.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4b908bb7-8558-48d3-a5c2-94b9c59f5dac_isrForCrosstalkSources_946852_4.stderr parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7d9681a383b74039ff08ea6d8548fc3 with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5ea25877-9b7c-46ff-a5bc-3c318879bc61_isrForCrosstalkSources_946853_25.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5ea25877-9b7c-46ff-a5bc-3c318879bc61_isrForCrosstalkSources_946853_25.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 036f5884adff7460b8a0da2b7a684135 with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7772cb4-1418-4fbe-8f2a-9a6f2e68de31_isrForCrosstalkSources_946852_24.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e7772cb4-1418-4fbe-8f2a-9a6f2e68de31_isrForCrosstalkSources_946852_24.stderr parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a72197d0a65fa2099511d1b5a17b9a with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c7591870-5939-4e51-ab7f-41196c6161a8_isrForCrosstalkSources_946852_30.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/c7591870-5939-4e51-ab7f-41196c6161a8_isrForCrosstalkSources_946852_30.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23728097ef8252165f5296a6f5c60b2d with result from task 343 parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c8f2d01d-9af9-4674-9f91-07738176ea30_isrForCrosstalkSources_946957_61.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c8f2d01d-9af9-4674-9f91-07738176ea30_isrForCrosstalkSources_946957_61.stderr parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07397781a2f3c57fc169e840641b7427 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e655ad6-6c9f-4083-81c4-b1c5e4ecf9d8_isrForCrosstalkSources_946959_7.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9e655ad6-6c9f-4083-81c4-b1c5e4ecf9d8_isrForCrosstalkSources_946959_7.stderr parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a6504e96a3e9d7cb9af032a89307663 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c6e0ad0a-11e4-4e1b-b11f-9c0517dd2d04_isrForCrosstalkSources_946958_49.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c6e0ad0a-11e4-4e1b-b11f-9c0517dd2d04_isrForCrosstalkSources_946958_49.stderr parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f64ab621aec5fcd7f994020cb9c4ec with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8b20cdcc-ee33-4d49-8377-ac9d94751bb0_isrForCrosstalkSources_946853_46.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8b20cdcc-ee33-4d49-8377-ac9d94751bb0_isrForCrosstalkSources_946853_46.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9935376958d27995e276bd46be9c9e1c with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9ca9789a-e574-4c1f-9a9c-cf144fcfb4e4_isrForCrosstalkSources_946959_27.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/9ca9789a-e574-4c1f-9a9c-cf144fcfb4e4_isrForCrosstalkSources_946959_27.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f19e2bea7771425d961deb2a571f413 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a37ae64c-a732-4fb7-8ae0-a896b474a028_isrForCrosstalkSources_946959_54.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/a37ae64c-a732-4fb7-8ae0-a896b474a028_isrForCrosstalkSources_946959_54.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fb4570dde8838ee61b345a95a6a918d with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/af7305ca-5da0-4ef4-96ea-5331c9153e3a_isrForCrosstalkSources_946852_60.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/af7305ca-5da0-4ef4-96ea-5331c9153e3a_isrForCrosstalkSources_946852_60.stderr parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaf964ed32350c9ad9ced7403809e8a7 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/cfc60417-d761-4b3c-b091-30da184a4bf7_isrForCrosstalkSources_946958_46.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/cfc60417-d761-4b3c-b091-30da184a4bf7_isrForCrosstalkSources_946958_46.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d79cf683586a94d8b539ca131c4b582f with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b26ccaf6-be69-46ab-9c82-62e4667c8769_isrForCrosstalkSources_946852_41.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b26ccaf6-be69-46ab-9c82-62e4667c8769_isrForCrosstalkSources_946852_41.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 973db377d346dc3ec06f21f36f5aaca0 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/08c4a4d3-26e5-424d-b58c-7f084a2819fc_isrForCrosstalkSources_946959_32.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/08c4a4d3-26e5-424d-b58c-7f084a2819fc_isrForCrosstalkSources_946959_32.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1be015f6ec3618e98801bf5e84440a1 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b0888412-a589-477b-9052-ef1c3faac0b8_isrForCrosstalkSources_946853_61.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b0888412-a589-477b-9052-ef1c3faac0b8_isrForCrosstalkSources_946853_61.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af49b279b52a995bfdb5d2c327cb14c3 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/a319e387-e62d-4f05-a2d9-821a601b6423_isrForCrosstalkSources_946957_35.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/a319e387-e62d-4f05-a2d9-821a601b6423_isrForCrosstalkSources_946957_35.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69d0400c373a9ee2c297f30afddd10a0 with result from task 289 parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/d32d636f-f29f-4149-8e4d-af367fc2625d_isrForCrosstalkSources_946958_29.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/d32d636f-f29f-4149-8e4d-af367fc2625d_isrForCrosstalkSources_946958_29.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df53caeed8bf3222a9945eea0c8417be with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee2e9f16-0577-4540-ad2c-6a96f319da86_isrForCrosstalkSources_946853_41.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ee2e9f16-0577-4540-ad2c-6a96f319da86_isrForCrosstalkSources_946853_41.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3de7e90e1ff2f9bf93e9a455b844a8fd with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/db115dfd-73d3-48cb-915d-04fa70d475f9_isrForCrosstalkSources_946959_39.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/db115dfd-73d3-48cb-915d-04fa70d475f9_isrForCrosstalkSources_946959_39.stderr parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ea548c9ef9d2a08f56b7236e7ca73ea with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/820c3f14-2ed7-4968-9a80-1de8bf8578f7_isrForCrosstalkSources_946851_9.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/820c3f14-2ed7-4968-9a80-1de8bf8578f7_isrForCrosstalkSources_946851_9.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 648dd18a32b4ebf2adfe6344bff67d7a with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/361e8f7c-bb42-41ac-b832-5869dd41bf51_isrForCrosstalkSources_946851_47.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/361e8f7c-bb42-41ac-b832-5869dd41bf51_isrForCrosstalkSources_946851_47.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7092224fced587fe05d47f717f456112 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3ecbd25c-e07c-4cde-be61-96d67256fde2_isrForCrosstalkSources_946957_43.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3ecbd25c-e07c-4cde-be61-96d67256fde2_isrForCrosstalkSources_946957_43.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1979514077b4a118a271937d37df8cb9 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a053c46d-c01f-4725-904c-cb0cba986b12_isrForCrosstalkSources_946956_4.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/a053c46d-c01f-4725-904c-cb0cba986b12_isrForCrosstalkSources_946956_4.stderr parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6b717c34ec4ddb2266849ee90227427 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/779314b8-b34d-4678-84bb-e9288be820da_isrForCrosstalkSources_946959_18.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/779314b8-b34d-4678-84bb-e9288be820da_isrForCrosstalkSources_946959_18.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfeffc3bca68187cf39683fb84f2d4cd with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/eb254a2a-cdfd-489b-9457-a250873c5053_isrForCrosstalkSources_946959_5.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/eb254a2a-cdfd-489b-9457-a250873c5053_isrForCrosstalkSources_946959_5.stderr parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba1f5b46513795f698c4843529419d89 with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/67f7cb9e-61de-40cf-94f3-e002e3f966ce_isrForCrosstalkSources_946959_22.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/67f7cb9e-61de-40cf-94f3-e002e3f966ce_isrForCrosstalkSources_946959_22.stderr parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2dac86ab98671195ac5da711ea6805e with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb567bc9-5df9-46cc-a19d-4f2cbdd7eec5_isrForCrosstalkSources_946957_4.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/fb567bc9-5df9-46cc-a19d-4f2cbdd7eec5_isrForCrosstalkSources_946957_4.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fd9320cb1d944da1bcdb3a4e7610fa6 with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/08dd767b-1655-4c9b-9883-83b1f5f99883_isrForCrosstalkSources_946957_20.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/08dd767b-1655-4c9b-9883-83b1f5f99883_isrForCrosstalkSources_946957_20.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 532c99b3142757a9bd083c4d15d539f3 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b8ea3e63-7512-40ed-8d45-da2a9b1b55ef_isrForCrosstalkSources_946853_27.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/b8ea3e63-7512-40ed-8d45-da2a9b1b55ef_isrForCrosstalkSources_946853_27.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 227524316b877f583da0ee3f6666b318 with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/048c10cc-c0ec-4721-9c4f-694d9475c468_isrForCrosstalkSources_946957_27.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/048c10cc-c0ec-4721-9c4f-694d9475c468_isrForCrosstalkSources_946957_27.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75d1c5733515be263c3a7532d3b5c53b with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5d94eced-766a-430a-9a21-67921674a3fb_isrForCrosstalkSources_946956_36.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5d94eced-766a-430a-9a21-67921674a3fb_isrForCrosstalkSources_946956_36.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b703ae51485f60760423dca45bb5a04c with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9374235a-1365-40f1-b367-c7eea7b9c5ae_isrForCrosstalkSources_946852_9.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9374235a-1365-40f1-b367-c7eea7b9c5ae_isrForCrosstalkSources_946852_9.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7acc6b56dd80fa2b069f3754ab1b258 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/37fac77d-b146-4d9b-b57c-e66d2b6997be_isrForCrosstalkSources_946956_3.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/37fac77d-b146-4d9b-b57c-e66d2b6997be_isrForCrosstalkSources_946956_3.stderr parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f19ef819da7eca3b0460fb1a1bff5f7e with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b4e3de86-d011-445d-82e8-39631658d999_isrForCrosstalkSources_946957_18.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/b4e3de86-d011-445d-82e8-39631658d999_isrForCrosstalkSources_946957_18.stderr parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4e975b2ce88ff58f715ef849b5fca63 with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/69a9c531-acc5-44e2-ba9b-c38ff307e616_isrForCrosstalkSources_946851_61.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/69a9c531-acc5-44e2-ba9b-c38ff307e616_isrForCrosstalkSources_946851_61.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9412892479cb66522d5a4128bf7cd16e with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/715afa1f-edd7-4fea-b1f2-bcabe0cf0545_isrForCrosstalkSources_946958_47.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/715afa1f-edd7-4fea-b1f2-bcabe0cf0545_isrForCrosstalkSources_946958_47.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4624efff7a60d18d6102e51a771fedc2 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/376efe91-a439-4e3d-96ec-697ccf0562b7_isrForCrosstalkSources_946956_58.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/376efe91-a439-4e3d-96ec-697ccf0562b7_isrForCrosstalkSources_946956_58.stderr parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 497efb4518f6b94fdb90170ad29d9775 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f4ddb1e7-83a7-4f19-b552-f1e150a753ad_isrForCrosstalkSources_946958_3.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/f4ddb1e7-83a7-4f19-b552-f1e150a753ad_isrForCrosstalkSources_946958_3.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1f3be6cbf09903dfa8eb42ab5828013 with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/64d3d327-ddc2-4239-83f6-ade4745972c5_isrForCrosstalkSources_946958_38.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/64d3d327-ddc2-4239-83f6-ade4745972c5_isrForCrosstalkSources_946958_38.stderr parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d70fb38a03efdc0c5c647320b05bd479 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4f1131c2-cc14-4ddb-ab95-a6b6b1bd1573_isrForCrosstalkSources_946959_14.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/4f1131c2-cc14-4ddb-ab95-a6b6b1bd1573_isrForCrosstalkSources_946959_14.stderr parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c791819e3f95d4de1608cf7757ee1fd9 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/33ad52c6-3516-4afb-9971-f14a28e804a1_isrForCrosstalkSources_946957_1.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/33ad52c6-3516-4afb-9971-f14a28e804a1_isrForCrosstalkSources_946957_1.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b265695f7a0b0b5ae8f43f0d80b46f15 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/a1d3bfb2-e201-4ade-9278-7fb1de8f855f_isrForCrosstalkSources_946852_56.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/a1d3bfb2-e201-4ade-9278-7fb1de8f855f_isrForCrosstalkSources_946852_56.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dfcaa6dd93e47f36d99689f50c14c3c with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2cefba49-f490-4161-b569-28f9ee35ca5d_isrForCrosstalkSources_946853_50.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2cefba49-f490-4161-b569-28f9ee35ca5d_isrForCrosstalkSources_946853_50.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 472564c5f7897233934e9240daf8781d with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b6740a67-2751-492f-bf09-fc53dbbb22d8_isrForCrosstalkSources_946958_10.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/b6740a67-2751-492f-bf09-fc53dbbb22d8_isrForCrosstalkSources_946958_10.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1b2ec56d67eb542adccedc01220f228 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd661873-6bf2-4059-aac0-df8294a0e189_isrForCrosstalkSources_946852_19.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd661873-6bf2-4059-aac0-df8294a0e189_isrForCrosstalkSources_946852_19.stderr parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55e6668f740e6d3ce8e8682bb93e2a69 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1749e71b-642c-48d1-8415-8c1a3cd46578_isrForCrosstalkSources_946958_1.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1749e71b-642c-48d1-8415-8c1a3cd46578_isrForCrosstalkSources_946958_1.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7301588d1a536f23f13fdd42199003f with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5577ee81-0a74-49cb-af68-0f752079f647_isrForCrosstalkSources_946853_42.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5577ee81-0a74-49cb-af68-0f752079f647_isrForCrosstalkSources_946853_42.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34888310810bc98585bbc0cffedeb2ce with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/846c8183-b95f-499e-a3ef-cdbd432a86e6_isrForCrosstalkSources_946852_51.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/846c8183-b95f-499e-a3ef-cdbd432a86e6_isrForCrosstalkSources_946852_51.stderr parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9db31740f2be12b6995ca1354a22d52 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/087b00ad-9885-49bb-bf9c-55189384c020_isrForCrosstalkSources_946959_25.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/087b00ad-9885-49bb-bf9c-55189384c020_isrForCrosstalkSources_946959_25.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4f584fc470402df20f461cef7599593 with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e09466e3-2bdb-4bd2-a279-f029b35d5f21_isrForCrosstalkSources_946957_25.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e09466e3-2bdb-4bd2-a279-f029b35d5f21_isrForCrosstalkSources_946957_25.stderr parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0649cad19cca3091250c9d78b141213 with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/5aa24502-766a-4321-a079-48617c8414e0_isrForCrosstalkSources_946959_49.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/5aa24502-766a-4321-a079-48617c8414e0_isrForCrosstalkSources_946959_49.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c9f337def735a47990661a4d0ec9e3e6 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/110d54ec-deee-4a4b-b267-254f6eb2fb83_isrForCrosstalkSources_946956_55.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/110d54ec-deee-4a4b-b267-254f6eb2fb83_isrForCrosstalkSources_946956_55.stderr parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6c8c37381338c25381e59d6e8c07c9d6 with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4658087a-495c-4ff0-b460-76d66e0d5e65_isrForCrosstalkSources_946958_31.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4658087a-495c-4ff0-b460-76d66e0d5e65_isrForCrosstalkSources_946958_31.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8cf3f319e5d4418e17f437dc5215d3c with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/82ba8888-76e6-4af8-ab06-8ad90d24760d_isrForCrosstalkSources_946851_22.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/82ba8888-76e6-4af8-ab06-8ad90d24760d_isrForCrosstalkSources_946851_22.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 726e103e1cd7c6bff6add802ec21a15c with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bada5e64-3cf5-43d7-a366-c8811ce316f6_isrForCrosstalkSources_946852_27.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/bada5e64-3cf5-43d7-a366-c8811ce316f6_isrForCrosstalkSources_946852_27.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0da2688ba4aaeed020cc88cc0d19128e with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/50c8144e-4c03-4ffe-8e99-ecff6c99b5a8_isrForCrosstalkSources_946852_46.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/50c8144e-4c03-4ffe-8e99-ecff6c99b5a8_isrForCrosstalkSources_946852_46.stderr parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a9bd60fe8a8cf059b3e8f972fba23a2 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/98a56cf1-a272-4ed1-8063-ae221be1deea_isrForCrosstalkSources_946853_19.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/98a56cf1-a272-4ed1-8063-ae221be1deea_isrForCrosstalkSources_946853_19.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1b7bda97db8f59c63c8f2b3849f28525 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7094e88e-0e75-4d7f-bab0-43a83146198f_isrForCrosstalkSources_946956_33.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7094e88e-0e75-4d7f-bab0-43a83146198f_isrForCrosstalkSources_946956_33.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2187829f264d6df71632c79b47bc5383 with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d0423ace-40ad-4c80-92af-67d06e77f151_isrForCrosstalkSources_946959_33.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/d0423ace-40ad-4c80-92af-67d06e77f151_isrForCrosstalkSources_946959_33.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22bc6288f538db5162d5bfbdcc051f75 with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/dadfc78e-9c35-4178-a494-c45015a2a1aa_isrForCrosstalkSources_946957_21.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/dadfc78e-9c35-4178-a494-c45015a2a1aa_isrForCrosstalkSources_946957_21.stderr parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a03239f471ce8125f40e804fc9cbe7a4 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/962b5253-cb76-4a5f-94f5-78f3d13b528c_isrForCrosstalkSources_946851_18.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/962b5253-cb76-4a5f-94f5-78f3d13b528c_isrForCrosstalkSources_946851_18.stderr parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26c2f01b417a272b85400960894723eb with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6ff31201-4a57-4cd4-9be1-72ddd013b6e4_isrForCrosstalkSources_946959_23.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6ff31201-4a57-4cd4-9be1-72ddd013b6e4_isrForCrosstalkSources_946959_23.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c2b5a2222db453d4afb41c66c289c42 with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1462ca36-2da0-4205-968e-ce63c8a66dbc_isrForCrosstalkSources_946959_62.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1462ca36-2da0-4205-968e-ce63c8a66dbc_isrForCrosstalkSources_946959_62.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 476b2b99111325e01d40058f4ff94ff6 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5e768e35-f34d-443f-b684-df554aae11e4_isrForCrosstalkSources_946956_42.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/5e768e35-f34d-443f-b684-df554aae11e4_isrForCrosstalkSources_946956_42.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec3c863f2dbd0e02df8cc1d820d2f96f with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bafc38b7-c930-4085-be7c-b04b11911fa5_isrForCrosstalkSources_946957_57.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/bafc38b7-c930-4085-be7c-b04b11911fa5_isrForCrosstalkSources_946957_57.stderr parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1804bbcae60a82f078ffc0667ef7d99a with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/85328e7a-3c4a-4f82-a7cb-88d5fa8707c7_isrForCrosstalkSources_946957_48.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/85328e7a-3c4a-4f82-a7cb-88d5fa8707c7_isrForCrosstalkSources_946957_48.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a412c083e6032ad1e75a00f57f9d3003 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/caae8d33-7ae9-413c-966b-714c76c0f9f3_isrForCrosstalkSources_946959_35.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/caae8d33-7ae9-413c-966b-714c76c0f9f3_isrForCrosstalkSources_946959_35.stderr parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee275c6e0273175c1433d801fbd00ab2 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7d03beb3-3ce5-481b-af4a-cb93931e2267_isrForCrosstalkSources_946956_31.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/7d03beb3-3ce5-481b-af4a-cb93931e2267_isrForCrosstalkSources_946956_31.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 13571cda347884098550cc2a169ee676 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6cc5dc0b-fbf6-4623-86b5-2fe371089ab2_isrForCrosstalkSources_946957_26.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/6cc5dc0b-fbf6-4623-86b5-2fe371089ab2_isrForCrosstalkSources_946957_26.stderr parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8d0b82aa2702979f4162dd7ee097cfd with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3db88200-0c2c-458d-8743-613252096777_isrForCrosstalkSources_946957_23.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/3db88200-0c2c-458d-8743-613252096777_isrForCrosstalkSources_946957_23.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 30a5ce8710e16abefdc1a2a11f09a51a with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/66548884-493b-4886-95a2-dc24d0db2f99_isrForCrosstalkSources_946958_40.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/66548884-493b-4886-95a2-dc24d0db2f99_isrForCrosstalkSources_946958_40.stderr parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcf889924c3ca079212bd9c010ca093f with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7a180154-c436-442a-8956-e3489db13a7e_isrForCrosstalkSources_946958_7.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/7a180154-c436-442a-8956-e3489db13a7e_isrForCrosstalkSources_946958_7.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b40471721b9128a03fc389a7bf479613 with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1aced3a3-4172-42ee-90b2-9859ec41ac76_isrForCrosstalkSources_946959_11.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/1aced3a3-4172-42ee-90b2-9859ec41ac76_isrForCrosstalkSources_946959_11.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0df018c4140f4b51b21f77096afb32a1 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/24320557-8dfc-4680-a8df-4b2a1888fff2_isrForCrosstalkSources_946956_30.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/24320557-8dfc-4680-a8df-4b2a1888fff2_isrForCrosstalkSources_946956_30.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8456b80672841713322b63d8c82733a2 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/9cfe5e25-ace1-4285-b080-9087dff34f79_isrForCrosstalkSources_946958_39.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/9cfe5e25-ace1-4285-b080-9087dff34f79_isrForCrosstalkSources_946958_39.stderr parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f866bcb4c4eda07556885b655b127e0 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c069fe90-e79a-43cd-9545-e108ef6fb53c_isrForCrosstalkSources_946959_1.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c069fe90-e79a-43cd-9545-e108ef6fb53c_isrForCrosstalkSources_946959_1.stderr parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8af7c4665fa38362cbfc2d9ce6cd216 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5857b11d-ff92-454b-9a38-2051ad1fd8f1_isrForCrosstalkSources_946957_10.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5857b11d-ff92-454b-9a38-2051ad1fd8f1_isrForCrosstalkSources_946957_10.stderr parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 302927eee1fdd55f800d170f2d173b3a with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae0ef6f8-e180-422a-b710-c5a4f59899e1_isrForCrosstalkSources_946853_28.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ae0ef6f8-e180-422a-b710-c5a4f59899e1_isrForCrosstalkSources_946853_28.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5105b8eadcfafd1c95864d195e8f148f with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1c85ed03-eea1-48fb-9dca-16c4b6556356_isrForCrosstalkSources_946957_9.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/1c85ed03-eea1-48fb-9dca-16c4b6556356_isrForCrosstalkSources_946957_9.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40c44f100ea9fc836a6d4b6e5d95f18 with result from task 196 parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a43ad8fd-b277-4081-a433-cd67f0399f35_isrForCrosstalkSources_946853_38.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/a43ad8fd-b277-4081-a433-cd67f0399f35_isrForCrosstalkSources_946853_38.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ef5ea20d63627c0c85455d1560d3452 with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8118b22c-156d-4543-9fc5-b11958cdd0df_isrForCrosstalkSources_946957_30.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8118b22c-156d-4543-9fc5-b11958cdd0df_isrForCrosstalkSources_946957_30.stderr parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056ccb9e61e33a21362ed171dfd51020 with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/338b829e-7d49-4873-b4b4-e547f10f7789_isrForCrosstalkSources_946957_19.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/338b829e-7d49-4873-b4b4-e547f10f7789_isrForCrosstalkSources_946957_19.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfbdf91593b00eb3c2bc625a4e037775 with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/28127eb4-36d3-443c-bbfa-692094563955_isrForCrosstalkSources_946853_52.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/28127eb4-36d3-443c-bbfa-692094563955_isrForCrosstalkSources_946853_52.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 089806ee35df190af95c001ee06218fc with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/867afdf3-407b-4e67-a6bc-a5af8c55e7e7_isrForCrosstalkSources_946956_43.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/867afdf3-407b-4e67-a6bc-a5af8c55e7e7_isrForCrosstalkSources_946956_43.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7795b17aea894089bcb943af4362686f with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3055acf6-3741-4231-986d-c63bc0628189_isrForCrosstalkSources_946851_58.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/3055acf6-3741-4231-986d-c63bc0628189_isrForCrosstalkSources_946851_58.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ebb248b8ef230ba6a8a68b4086c106 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/358584a4-154d-4536-a01e-2e438fae818d_isrForCrosstalkSources_946851_28.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/358584a4-154d-4536-a01e-2e438fae818d_isrForCrosstalkSources_946851_28.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5581e0f27b0298c231b5a448910b5068 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d60b2f19-9df4-4e85-b6f4-e7b9a3963f04_isrForCrosstalkSources_946851_57.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/d60b2f19-9df4-4e85-b6f4-e7b9a3963f04_isrForCrosstalkSources_946851_57.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d9b6e6a20abb99eb656710c115e70b7 with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b6705144-ca81-4727-96f4-29c22058d9b1_isrForCrosstalkSources_946851_7.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/b6705144-ca81-4727-96f4-29c22058d9b1_isrForCrosstalkSources_946851_7.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 235834a37676cde97370220a1a9e94ff with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c7cd7a0b-f3cb-48d6-b3f8-d061115f8c7e_isrForCrosstalkSources_946957_36.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/c7cd7a0b-f3cb-48d6-b3f8-d061115f8c7e_isrForCrosstalkSources_946957_36.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d1f50cf5d366c21b4d4fcc33761baea with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ca13772c-69d3-48f1-971a-6cb9caed536c_isrForCrosstalkSources_946853_48.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/ca13772c-69d3-48f1-971a-6cb9caed536c_isrForCrosstalkSources_946853_48.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8cf136f6d5ce6b3508eb73697dc2cc3f with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1c870778-e43c-4ab9-a8dc-f71c153d7d7a_isrForCrosstalkSources_946958_9.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1c870778-e43c-4ab9-a8dc-f71c153d7d7a_isrForCrosstalkSources_946958_9.stderr parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1faf5d0ac3ab82fdd861b56c9ba456b with result from task 189 parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/02eebfd4-70db-494f-931c-f6650ee258e2_isrForCrosstalkSources_946852_43.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/02eebfd4-70db-494f-931c-f6650ee258e2_isrForCrosstalkSources_946852_43.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7f5b6bcc388e2c8458754967a5e24a2 with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/bd1f7d15-6d67-4062-bfcf-fb7f4a6247df_isrForCrosstalkSources_946958_25.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/bd1f7d15-6d67-4062-bfcf-fb7f4a6247df_isrForCrosstalkSources_946958_25.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b468c8dace85f5292286372d59c4c23 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/fe1dfb03-ce83-4bc9-9251-435be52beca4_isrForCrosstalkSources_946851_13.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/fe1dfb03-ce83-4bc9-9251-435be52beca4_isrForCrosstalkSources_946851_13.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0898ba7695d0bd5207369da1844652f with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5668d15e-7341-43d5-a965-7f78ae3d4c59_isrForCrosstalkSources_946853_56.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/5668d15e-7341-43d5-a965-7f78ae3d4c59_isrForCrosstalkSources_946853_56.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 652eefb04112fa59fe7a55babe1558ff with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aa974448-c81a-4c6a-8dd7-e9bb33b3e7e6_isrForCrosstalkSources_946852_57.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aa974448-c81a-4c6a-8dd7-e9bb33b3e7e6_isrForCrosstalkSources_946852_57.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a10b6589269d3f6c08b6af380b94231b with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4cf1bb61-ac16-413a-bae0-fc09e811698a_isrForCrosstalkSources_946852_1.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/4cf1bb61-ac16-413a-bae0-fc09e811698a_isrForCrosstalkSources_946852_1.stderr parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32a2887ac9d1c8cebb1743d8b3a90876 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd06bf96-ac70-4ba7-aeb7-608469cc1dd5_isrForCrosstalkSources_946852_49.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/cd06bf96-ac70-4ba7-aeb7-608469cc1dd5_isrForCrosstalkSources_946852_49.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4865a5fe8903fa8b5833fcc6281fafc7 with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a8a939-1256-4b99-bd85-8169eb6cce68_isrForCrosstalkSources_946852_12.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a8a939-1256-4b99-bd85-8169eb6cce68_isrForCrosstalkSources_946852_12.stderr parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e08649003493a44ff1341d8c7afc8ffa with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aeecd840-7601-4352-804a-4a3b7e07d934_isrForCrosstalkSources_946852_48.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/aeecd840-7601-4352-804a-4a3b7e07d934_isrForCrosstalkSources_946852_48.stderr parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68d7bf25624b21a7409765d957346212 with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4e6746e1-3667-4d19-9df9-6503b7bf33c2_isrForCrosstalkSources_946851_42.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4e6746e1-3667-4d19-9df9-6503b7bf33c2_isrForCrosstalkSources_946851_42.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b3d72e0b3ab66a2435d19e409726be7 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/76b6803b-e52f-40df-9708-c30f41948588_isrForCrosstalkSources_946851_12.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/76b6803b-e52f-40df-9708-c30f41948588_isrForCrosstalkSources_946851_12.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f29727a06706ee37244a3439bee6b0a8 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4135553a-e573-4be6-983a-22604b888d62_isrForCrosstalkSources_946851_20.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/4135553a-e573-4be6-983a-22604b888d62_isrForCrosstalkSources_946851_20.stderr parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8fb0bce0621f4293cacddc1cdc0e17d with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/12d3bc93-b88f-4619-97c7-e852d2933cc2_isrForCrosstalkSources_946958_58.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/12d3bc93-b88f-4619-97c7-e852d2933cc2_isrForCrosstalkSources_946958_58.stderr 2024-03-14 04:34:19 proc_lsst.multi:146 [INFO] found job 98211 in provider local proc_lsst.multi INFO: found job 98211 in provider local 2024-03-14 04:34:19 proc_lsst.multi:146 [INFO] found job 17030162 in provider astro proc_lsst.multi INFO: found job 17030162 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:19 proc_lsst.multi:146 [INFO] found job 17030163 in provider astro proc_lsst.multi INFO: found job 17030163 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:19 proc_lsst.multi:146 [INFO] found job 17030164 in provider astro proc_lsst.multi INFO: found job 17030164 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:19 proc_lsst.multi:146 [INFO] found job 17030165 in provider astro proc_lsst.multi INFO: found job 17030165 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030166 in provider astro proc_lsst.multi INFO: found job 17030166 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030167 in provider astro proc_lsst.multi INFO: found job 17030167 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030168 in provider astro proc_lsst.multi INFO: found job 17030168 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030169 in provider astro proc_lsst.multi INFO: found job 17030169 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030170 in provider astro proc_lsst.multi INFO: found job 17030170 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030171 in provider astro proc_lsst.multi INFO: found job 17030171 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030172 in provider astro proc_lsst.multi INFO: found job 17030172 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:20 proc_lsst.multi:146 [INFO] found job 17030173 in provider astro proc_lsst.multi INFO: found job 17030173 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030174 in provider astro proc_lsst.multi INFO: found job 17030174 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030175 in provider astro proc_lsst.multi INFO: found job 17030175 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030176 in provider astro proc_lsst.multi INFO: found job 17030176 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030177 in provider astro proc_lsst.multi INFO: found job 17030177 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030178 in provider astro proc_lsst.multi INFO: found job 17030178 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030179 in provider astro proc_lsst.multi INFO: found job 17030179 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030180 in provider astro proc_lsst.multi INFO: found job 17030180 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030181 in provider astro proc_lsst.multi INFO: found job 17030181 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:21 proc_lsst.multi:146 [INFO] found job 17030182 in provider astro proc_lsst.multi INFO: found job 17030182 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030183 in provider astro proc_lsst.multi INFO: found job 17030183 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030184 in provider astro proc_lsst.multi INFO: found job 17030184 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030185 in provider astro proc_lsst.multi INFO: found job 17030185 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030186 in provider astro proc_lsst.multi INFO: found job 17030186 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030187 in provider astro proc_lsst.multi INFO: found job 17030187 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030188 in provider astro proc_lsst.multi INFO: found job 17030188 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030189 in provider astro proc_lsst.multi INFO: found job 17030189 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:22 proc_lsst.multi:146 [INFO] found job 17030190 in provider astro proc_lsst.multi INFO: found job 17030190 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030191 in provider astro proc_lsst.multi INFO: found job 17030191 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030162,17030163,17030164,17030165,17030166,17030167,17030168,17030169,17030170,17030171,17030172,17030173,17030174,17030175,17030176,17030177,17030178,17030179,17030180,17030181,17030182,17030183,17030184,17030185,17030186,17030187,17030188,17030189,17030190,17030191' parsl.providers.slurm.slurm DEBUG: squeue returned 17030184 R 17030185 R 17030186 R 17030187 R 17030188 R 17030189 R 17030190 R 17030191 R 17030174 R 17030175 R 17030176 R 17030177 R 17030178 R 17030179 R 17030180 R 17030181 R 17030182 R 17030183 R 17030165 R 17030166 R 17030167 R 17030168 R 17030169 R 17030170 R 17030171 R 17030172 R 17030173 R 17030162 R 17030163 R 17030164 R parsl.providers.slurm.slurm DEBUG: Updating job 17030184 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030185 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030186 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030187 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030188 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030189 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030190 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030191 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030174 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030175 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030176 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030177 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030178 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030179 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030180 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030181 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030182 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030183 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030165 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030166 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030167 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030168 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030169 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030170 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030171 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030172 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030173 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030162 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030163 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030164 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030192 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030192 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030193 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030193 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030194 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030194 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030195 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030195 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030196 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030196 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030192,17030193,17030194,17030195,17030196' parsl.providers.slurm.slurm DEBUG: squeue returned 17030192 R 17030193 R 17030194 R 17030195 R 17030196 R parsl.providers.slurm.slurm DEBUG: Updating job 17030192 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030193 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030194 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030195 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030196 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030197 in provider escience_a40 proc_lsst.multi INFO: found job 17030197 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030198 in provider escience_a40 proc_lsst.multi INFO: found job 17030198 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:23 proc_lsst.multi:146 [INFO] found job 17030199 in provider escience_a40 proc_lsst.multi INFO: found job 17030199 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030200 in provider escience_a40 proc_lsst.multi INFO: found job 17030200 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030201 in provider escience_a40 proc_lsst.multi INFO: found job 17030201 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030202 in provider escience_a40 proc_lsst.multi INFO: found job 17030202 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030203 in provider escience_a40 proc_lsst.multi INFO: found job 17030203 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030204 in provider escience_a40 proc_lsst.multi INFO: found job 17030204 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030205 in provider escience_a40 proc_lsst.multi INFO: found job 17030205 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030206 in provider escience_a40 proc_lsst.multi INFO: found job 17030206 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:24 proc_lsst.multi:146 [INFO] found job 17030207 in provider escience_a40 proc_lsst.multi INFO: found job 17030207 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030208 in provider escience_a40 proc_lsst.multi INFO: found job 17030208 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030209 in provider escience_a40 proc_lsst.multi INFO: found job 17030209 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030210 in provider escience_a40 proc_lsst.multi INFO: found job 17030210 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030211 in provider escience_a40 proc_lsst.multi INFO: found job 17030211 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030213 in provider escience_a40 proc_lsst.multi INFO: found job 17030213 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030214 in provider escience_a40 proc_lsst.multi INFO: found job 17030214 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030215 in provider escience_a40 proc_lsst.multi INFO: found job 17030215 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030216 in provider escience_a40 proc_lsst.multi INFO: found job 17030216 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:25 proc_lsst.multi:146 [INFO] found job 17030217 in provider escience_a40 proc_lsst.multi INFO: found job 17030217 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030197,17030198,17030199,17030200,17030201,17030202,17030203,17030204,17030205,17030206,17030207,17030208,17030209,17030210,17030211,17030213,17030214,17030215,17030216,17030217' parsl.providers.slurm.slurm DEBUG: squeue returned 17030216 R 17030217 R 17030209 R 17030210 R 17030211 R 17030213 R 17030214 R 17030215 R 17030199 R 17030200 R 17030201 R 17030202 R 17030203 R 17030204 R 17030205 R 17030206 R 17030207 R 17030208 R 17030197 R 17030198 R parsl.providers.slurm.slurm DEBUG: Updating job 17030216 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030217 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030209 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030210 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030211 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030213 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030214 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030215 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030199 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030200 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030201 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030202 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030203 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030204 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030205 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030206 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030207 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030208 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030197 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030198 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030218 in provider checkpoint proc_lsst.multi INFO: found job 17030218 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030219 in provider checkpoint proc_lsst.multi INFO: found job 17030219 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e56fb8be82951647d90d2700e4024177 with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/29952163-f276-4268-af32-e62c41be5aee_isrForCrosstalkSources_946957_8.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/29952163-f276-4268-af32-e62c41be5aee_isrForCrosstalkSources_946957_8.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 538906f21711d7ad825830d2abd2204d with result from task 379 parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030220 in provider checkpoint proc_lsst.multi INFO: found job 17030220 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b021f934-9ad9-4972-9907-292d9c4bdadb_isrForCrosstalkSources_946956_25.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b021f934-9ad9-4972-9907-292d9c4bdadb_isrForCrosstalkSources_946956_25.stderr parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 919cbd4240e8b9493b6d8759ceecca07 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b9a6545f-f17c-4245-b6e3-f197ac0c943f_isrForCrosstalkSources_946956_2.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/b9a6545f-f17c-4245-b6e3-f197ac0c943f_isrForCrosstalkSources_946956_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030221 in provider checkpoint proc_lsst.multi INFO: found job 17030221 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ecf52fcdf01b972384d0616396d2e857 with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/f35a74bb-2c60-4293-984e-38268aa281b7_isrForCrosstalkSources_946956_28.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/f35a74bb-2c60-4293-984e-38268aa281b7_isrForCrosstalkSources_946956_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry e4ec8f489193c276ad6c901f8bde3b30 with result from task 351 parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030222 in provider checkpoint proc_lsst.multi INFO: found job 17030222 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb31f737-d90a-4c2f-9ec9-956ccf9d4c6b_isrForCrosstalkSources_946851_4.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/eb31f737-d90a-4c2f-9ec9-956ccf9d4c6b_isrForCrosstalkSources_946851_4.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3bd9409589178694e70c698c01ace67 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a799e0-58b3-49ef-9984-6f3a952836a1_isrForCrosstalkSources_946852_14.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/61a799e0-58b3-49ef-9984-6f3a952836a1_isrForCrosstalkSources_946852_14.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0dc6f0e7db7fb4941d42fcebf9e14332 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/9507c6e7-a1d2-42e2-bd88-e5810bbae759_isrForCrosstalkSources_946957_41.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/9507c6e7-a1d2-42e2-bd88-e5810bbae759_isrForCrosstalkSources_946957_41.stderr parsl.dataflow.dflow INFO: Task 370 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6efb877d0583214ecd99f706eace54b6 with result from task 370 parsl.dataflow.dflow DEBUG: Task 370 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/1f5e2513-9889-4d93-b5c7-5d7de14c49b4_isrForCrosstalkSources_946852_34.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/1f5e2513-9889-4d93-b5c7-5d7de14c49b4_isrForCrosstalkSources_946852_34.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e22e04df56313dd088088e45ceea5101 with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/10ab4408-6265-48ba-899a-eca786ece055_isrForCrosstalkSources_946958_21.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/10ab4408-6265-48ba-899a-eca786ece055_isrForCrosstalkSources_946958_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 0c47c1fcd0ae07231ca5d9eef1bc9acf with result from task 381 parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030223 in provider checkpoint proc_lsst.multi INFO: found job 17030223 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/390db2e5-3c94-4048-bfb2-629e39c8fc49_isrForCrosstalkSources_946958_36.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/390db2e5-3c94-4048-bfb2-629e39c8fc49_isrForCrosstalkSources_946958_36.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b083e2381f7f1d039ddebb838fd98318 with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ab66a1d5-ce0a-4a3e-ae77-d6d362108406_isrForCrosstalkSources_946956_62.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ab66a1d5-ce0a-4a3e-ae77-d6d362108406_isrForCrosstalkSources_946956_62.stderr parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c1b9c52abbf3e35ef6a3636ffd7c5e7 with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c686bed8-e74b-4eba-b201-2aa2d29418a5_isrForCrosstalkSources_946956_40.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/c686bed8-e74b-4eba-b201-2aa2d29418a5_isrForCrosstalkSources_946956_40.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 329e6578c2f348ae3a128313aa6689f3 with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1b711e61-8cc6-46c3-a9b6-e44694c3d88c_isrForCrosstalkSources_946958_45.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/1b711e61-8cc6-46c3-a9b6-e44694c3d88c_isrForCrosstalkSources_946958_45.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfdd810440c17fe6eef33b8583b97489 with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/87ef3393-ae95-4695-9ecf-0baf96a139e5_isrForCrosstalkSources_946958_12.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/87ef3393-ae95-4695-9ecf-0baf96a139e5_isrForCrosstalkSources_946958_12.stderr parsl.dataflow.dflow INFO: Task 371 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39eeb1c83c06f48ff23f292a2d03154e with result from task 371 parsl.dataflow.dflow DEBUG: Task 371 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4dca462b-0515-46d2-9a62-b1aaca840fb6_isrForCrosstalkSources_946958_34.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4dca462b-0515-46d2-9a62-b1aaca840fb6_isrForCrosstalkSources_946958_34.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85ae11cb759d9372e046c450cf9cb9e1 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ee1e1ce6-9205-4ef4-b7c3-73cb8d4bfe55_isrForCrosstalkSources_946957_3.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/ee1e1ce6-9205-4ef4-b7c3-73cb8d4bfe55_isrForCrosstalkSources_946957_3.stderr parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1701a18bf740536791a12a648e3a5e18 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b32b6dfd-2538-4b09-a80d-796d92b27e21_isrForCrosstalkSources_946852_15.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/b32b6dfd-2538-4b09-a80d-796d92b27e21_isrForCrosstalkSources_946852_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.memoization DEBUG: Storing app cache entry 56f529f6512798e33bafcdaba9cb555f with result from task 367 parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ca877b40-5ffd-461a-a30c-cdff1cd3fd4a_isrForCrosstalkSources_946956_56.stdout parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/ca877b40-5ffd-461a-a30c-cdff1cd3fd4a_isrForCrosstalkSources_946956_56.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030224 in provider checkpoint proc_lsst.multi INFO: found job 17030224 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d68843c617f8275e3632a116f10f0ac3 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/461f8a6f-4580-4fe9-9128-de365128a74b_isrForCrosstalkSources_946852_40.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/461f8a6f-4580-4fe9-9128-de365128a74b_isrForCrosstalkSources_946852_40.stderr parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 847a5888b7f39767952c6c73d7fff200 with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/55f98655-0d89-4cdb-ac45-1f100eb32744_isrForCrosstalkSources_946852_18.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/55f98655-0d89-4cdb-ac45-1f100eb32744_isrForCrosstalkSources_946852_18.stderr parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62ef6a52e228270b229f17f1003e317c with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/810bf6e5-a319-486e-a548-391c9677f2dd_isrForCrosstalkSources_946957_49.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/810bf6e5-a319-486e-a548-391c9677f2dd_isrForCrosstalkSources_946957_49.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2c52d506aa09b11759549d68cfc7236 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3683825d-2429-4cd0-adc2-377660b07024_isrForCrosstalkSources_946853_21.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/3683825d-2429-4cd0-adc2-377660b07024_isrForCrosstalkSources_946853_21.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e90a913db3a40094a76ebdeff133a7d with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/831da753-f95f-4b0b-88c8-093165cecdb9_isrForCrosstalkSources_946956_10.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/831da753-f95f-4b0b-88c8-093165cecdb9_isrForCrosstalkSources_946956_10.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4918f55792cbd43fe7866825c2051c7 with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a9f84d4-aac5-41f3-8afd-6395d8785eaf_isrForCrosstalkSources_946957_59.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/8a9f84d4-aac5-41f3-8afd-6395d8785eaf_isrForCrosstalkSources_946957_59.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8166e2bc631b872d40fe8fa65882396 with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/785f904e-c0a7-440f-be7c-6a4120b035b7_isrForCrosstalkSources_946957_42.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/785f904e-c0a7-440f-be7c-6a4120b035b7_isrForCrosstalkSources_946957_42.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34cf6c6fdd9a614cd79fca019b868a1d with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/1b9d65ef-c057-4ebe-9131-5a20e8d1f8c3_isrForCrosstalkSources_946853_16.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/1b9d65ef-c057-4ebe-9131-5a20e8d1f8c3_isrForCrosstalkSources_946853_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:26 proc_lsst.multi:146 [INFO] found job 17030225 in provider checkpoint proc_lsst.multi INFO: found job 17030225 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030226 in provider checkpoint proc_lsst.multi INFO: found job 17030226 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f564da81b846f7273ed57e216709c7a8 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/744b4cb9-c21a-4964-a604-cfa138084b40_isrForCrosstalkSources_946852_10.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/744b4cb9-c21a-4964-a604-cfa138084b40_isrForCrosstalkSources_946852_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030227 in provider checkpoint proc_lsst.multi INFO: found job 17030227 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db13ce2861c1e6e5624a985ddb59b06b with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/05c7f1eb-6e7b-4c82-986e-4e18aad0da79_isrForCrosstalkSources_946853_57.stdout parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/05c7f1eb-6e7b-4c82-986e-4e18aad0da79_isrForCrosstalkSources_946853_57.stderr parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030228 in provider checkpoint proc_lsst.multi INFO: found job 17030228 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030229 in provider checkpoint proc_lsst.multi INFO: found job 17030229 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ef5b5a8af86d9d677fc048d6705e7b0 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cba3c0de-bfef-459e-9079-132b53e94130_isrForCrosstalkSources_946853_47.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/cba3c0de-bfef-459e-9079-132b53e94130_isrForCrosstalkSources_946853_47.stderr parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2bb00ee059031bd4e1b5ef139c81a5c with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a3cb6b2-832b-400a-83ea-5c569c85ce67_isrForCrosstalkSources_946851_2.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0a3cb6b2-832b-400a-83ea-5c569c85ce67_isrForCrosstalkSources_946851_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030230 in provider checkpoint proc_lsst.multi INFO: found job 17030230 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c43551d7224d3156bd63f164f2693c71 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a4e1da3-b9db-482b-8eef-f7e87d1d3cf0_isrForCrosstalkSources_946853_53.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/9a4e1da3-b9db-482b-8eef-f7e87d1d3cf0_isrForCrosstalkSources_946853_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030231 in provider checkpoint proc_lsst.multi INFO: found job 17030231 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8d0dee3797a4dabbf14105a4c00d0e3 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/974a8879-f455-4522-8cab-3cac49738d8b_isrForCrosstalkSources_946852_36.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/974a8879-f455-4522-8cab-3cac49738d8b_isrForCrosstalkSources_946852_36.stderr parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a0bf92e73955d91a1ce2846d390b7d33 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/41488292-923f-4059-a3f6-49c5406d3be6_isrForCrosstalkSources_946852_29.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/41488292-923f-4059-a3f6-49c5406d3be6_isrForCrosstalkSources_946852_29.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7708189300df2ccd60ed280d4970d9de with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/69bb6e33-33f8-4bbc-b65f-1fc16965e0e3_isrForCrosstalkSources_946852_50.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/69bb6e33-33f8-4bbc-b65f-1fc16965e0e3_isrForCrosstalkSources_946852_50.stderr parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a48530ed60bcb69b6a19e6a691cb300 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/abc05599-cb31-4911-b954-8bd663144e37_isrForCrosstalkSources_946957_44.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/abc05599-cb31-4911-b954-8bd663144e37_isrForCrosstalkSources_946957_44.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2e1fd353567504d97581110b5099368 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6fbffe2f-bf76-4780-99f5-c4fbded9a2aa_isrForCrosstalkSources_946959_38.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/6fbffe2f-bf76-4780-99f5-c4fbded9a2aa_isrForCrosstalkSources_946959_38.stderr parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 575809104738186189e31c492249f274 with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2b826b9b-da24-496a-b199-fc0d6bf90b8f_isrForCrosstalkSources_946853_1.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/2b826b9b-da24-496a-b199-fc0d6bf90b8f_isrForCrosstalkSources_946853_1.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da6d734cf5998746f29119ad8e45f9af with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f3abb5c0-1bac-45ba-88cb-d3c82b8668d5_isrForCrosstalkSources_946853_10.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/f3abb5c0-1bac-45ba-88cb-d3c82b8668d5_isrForCrosstalkSources_946853_10.stderr parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e56036b394c825ca31599694ae61a7c6 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/1a15b574-23dd-4c3b-9eec-fcc03c198d02_isrForCrosstalkSources_946851_37.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/1a15b574-23dd-4c3b-9eec-fcc03c198d02_isrForCrosstalkSources_946851_37.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0235b7facde919efafe40fafd5ffc94 with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13b1648a-d7ef-4b0a-b3e6-e06e8645b6f8_isrForCrosstalkSources_946959_29.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/13b1648a-d7ef-4b0a-b3e6-e06e8645b6f8_isrForCrosstalkSources_946959_29.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da597c7d3c2648eb4170562488243413 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d78fd705-2e79-445d-9ff1-9cdd69d93c39_isrForCrosstalkSources_946956_37.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946956/d78fd705-2e79-445d-9ff1-9cdd69d93c39_isrForCrosstalkSources_946956_37.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29788f66d307e1627b8d368a5a9f1f32 with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/57a58161-5029-42e5-a000-1c3364f817c8_isrForCrosstalkSources_946853_31.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/57a58161-5029-42e5-a000-1c3364f817c8_isrForCrosstalkSources_946853_31.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66f3ba764761e4b4b673d90ed21737cd with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5318a02f-ad87-439d-b507-f116858cb21a_isrForCrosstalkSources_946957_62.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/5318a02f-ad87-439d-b507-f116858cb21a_isrForCrosstalkSources_946957_62.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bbefd754d8c47f12db0c983d80a8609e with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/fb40bd96-27dc-43b2-9edf-0664892c193d_isrForCrosstalkSources_946852_6.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/fb40bd96-27dc-43b2-9edf-0664892c193d_isrForCrosstalkSources_946852_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030232 in provider checkpoint proc_lsst.multi INFO: found job 17030232 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcfbe56b140d6bae3b5495ec0de8eeab with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0c7730d-9e74-4276-a540-bd125649c337_isrForCrosstalkSources_946852_37.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0c7730d-9e74-4276-a540-bd125649c337_isrForCrosstalkSources_946852_37.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ad6528038b5779fb1b9de2f5fb3684 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e0c978db-e370-4950-89f3-f179e6cba43e_isrForCrosstalkSources_946852_17.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/e0c978db-e370-4950-89f3-f179e6cba43e_isrForCrosstalkSources_946852_17.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd94dc3edb7c5bdb9af3d2d7474e9392 with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/37ca2676-dd03-4daa-a9da-85b923c8d9aa_isrForCrosstalkSources_946958_8.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/37ca2676-dd03-4daa-a9da-85b923c8d9aa_isrForCrosstalkSources_946958_8.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 716defbbfee079ba01226760515984e4 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c0e0e138-14ff-4b8b-96f7-b976527e428b_isrForCrosstalkSources_946959_21.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946959/c0e0e138-14ff-4b8b-96f7-b976527e428b_isrForCrosstalkSources_946959_21.stderr parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02ddbe59c23d748c03f8a583dfc53b16 with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0cc59197-a2a5-463c-8829-c72ba5062566_isrForCrosstalkSources_946851_19.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946851/0cc59197-a2a5-463c-8829-c72ba5062566_isrForCrosstalkSources_946851_19.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a320090ba799001fb17daf1bf3621c9 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/907df2be-38dd-4344-baad-8fedadcbd4e2_isrForCrosstalkSources_946852_62.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/907df2be-38dd-4344-baad-8fedadcbd4e2_isrForCrosstalkSources_946852_62.stderr parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3aeca8694f4553fa27c20e8806503d8c with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4436ee44-e1b0-4b58-a36a-f01c63dc0465_isrForCrosstalkSources_946958_26.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/4436ee44-e1b0-4b58-a36a-f01c63dc0465_isrForCrosstalkSources_946958_26.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 930d1e0d5de8d70b86c4a72eca389e0b with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e4eda809-bc5c-47b8-bf76-c68e37c7092e_isrForCrosstalkSources_946957_50.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946957/e4eda809-bc5c-47b8-bf76-c68e37c7092e_isrForCrosstalkSources_946957_50.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0048d808b17460b7f740db54b830491a with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c3a356e3-d982-4754-96ad-d71f17e4a461_isrForCrosstalkSources_946958_60.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946958/c3a356e3-d982-4754-96ad-d71f17e4a461_isrForCrosstalkSources_946958_60.stderr parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52a96061c9f0c91af5863eb28c17268d with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9dd49445-acf6-4553-b338-2a5caedadc61_isrForCrosstalkSources_946852_16.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/9dd49445-acf6-4553-b338-2a5caedadc61_isrForCrosstalkSources_946852_16.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3596b5f871d6e1bfcd05c1892f41dac6 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8804eec0-50d8-4d09-98b4-a43a493596da_isrForCrosstalkSources_946853_6.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946853/8804eec0-50d8-4d09-98b4-a43a493596da_isrForCrosstalkSources_946853_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:27 proc_lsst.multi:146 [INFO] found job 17030233 in provider checkpoint proc_lsst.multi INFO: found job 17030233 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 628d5dfad18574948c2672d1c409ff55 with result from task 384 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 434 parsl.dataflow.dflow INFO: Tasks in state States.failed: 0 parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 0 parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0fb3b41-519d-4f0d-adbe-808589428e02_isrForCrosstalkSources_946852_55.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/submit/DEEP/20201019/B1a/science#step0/20240314T043249Z/logs/isrForCrosstalkSources/946852/f0fb3b41-519d-4f0d-adbe-808589428e02_isrForCrosstalkSources_946852_55.stderr parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030234 in provider checkpoint proc_lsst.multi INFO: found job 17030234 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030235 in provider checkpoint proc_lsst.multi INFO: found job 17030235 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030236 in provider checkpoint proc_lsst.multi INFO: found job 17030236 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030237 in provider checkpoint proc_lsst.multi INFO: found job 17030237 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030238 in provider checkpoint proc_lsst.multi INFO: found job 17030238 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030239 in provider checkpoint proc_lsst.multi INFO: found job 17030239 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030240 in provider checkpoint proc_lsst.multi INFO: found job 17030240 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:28 proc_lsst.multi:146 [INFO] found job 17030241 in provider checkpoint proc_lsst.multi INFO: found job 17030241 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030242 in provider checkpoint proc_lsst.multi INFO: found job 17030242 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030243 in provider checkpoint proc_lsst.multi INFO: found job 17030243 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030244 in provider checkpoint proc_lsst.multi INFO: found job 17030244 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030245 in provider checkpoint proc_lsst.multi INFO: found job 17030245 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030246 in provider checkpoint proc_lsst.multi INFO: found job 17030246 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030247 in provider checkpoint proc_lsst.multi INFO: found job 17030247 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030248 in provider checkpoint proc_lsst.multi INFO: found job 17030248 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:29 proc_lsst.multi:146 [INFO] found job 17030249 in provider checkpoint proc_lsst.multi INFO: found job 17030249 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030250 in provider checkpoint proc_lsst.multi INFO: found job 17030250 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030251 in provider checkpoint proc_lsst.multi INFO: found job 17030251 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030252 in provider checkpoint proc_lsst.multi INFO: found job 17030252 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030253 in provider checkpoint proc_lsst.multi INFO: found job 17030253 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030254 in provider checkpoint proc_lsst.multi INFO: found job 17030254 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030255 in provider checkpoint proc_lsst.multi INFO: found job 17030255 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030257 in provider checkpoint proc_lsst.multi INFO: found job 17030257 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030258 in provider checkpoint proc_lsst.multi INFO: found job 17030258 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:30 proc_lsst.multi:146 [INFO] found job 17030259 in provider checkpoint proc_lsst.multi INFO: found job 17030259 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030260 in provider checkpoint proc_lsst.multi INFO: found job 17030260 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030261 in provider checkpoint proc_lsst.multi INFO: found job 17030261 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030262 in provider checkpoint proc_lsst.multi INFO: found job 17030262 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030263 in provider checkpoint proc_lsst.multi INFO: found job 17030263 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030264 in provider checkpoint proc_lsst.multi INFO: found job 17030264 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030265 in provider checkpoint proc_lsst.multi INFO: found job 17030265 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030266 in provider checkpoint proc_lsst.multi INFO: found job 17030266 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:31 proc_lsst.multi:146 [INFO] found job 17030267 in provider checkpoint proc_lsst.multi INFO: found job 17030267 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030268 in provider checkpoint proc_lsst.multi INFO: found job 17030268 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030269 in provider checkpoint proc_lsst.multi INFO: found job 17030269 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030270 in provider checkpoint proc_lsst.multi INFO: found job 17030270 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030271 in provider checkpoint proc_lsst.multi INFO: found job 17030271 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '17030218,17030219,17030220,17030221,17030222,17030223,17030224,17030225,17030226,17030227,17030228,17030229,17030230,17030231,17030232,17030233,17030234,17030235,17030236,17030237,17030238,17030239,17030240,17030241,17030242,17030243,17030244,17030245,17030246,17030247,17030248,17030249,17030250,17030251,17030252,17030253,17030254,17030255,17030257,17030258,17030259,17030260,17030261,17030262,17030263,17030264,17030265,17030266,17030267,17030268,17030269,17030270,17030271' parsl.providers.slurm.slurm DEBUG: squeue returned 17030267 R 17030268 R 17030269 R 17030270 R 17030271 R 17030257 R 17030258 R 17030259 R 17030260 R 17030261 R 17030262 R 17030263 R 17030264 R 17030265 R 17030266 R 17030248 R 17030249 R 17030250 R 17030251 R 17030252 R 17030253 R 17030254 R 17030255 R 17030238 R 17030239 R 17030240 R 17030241 R 17030242 R 17030243 R 17030244 R 17030245 R 17030246 R 17030247 R 17030231 R 17030232 R 17030233 R 17030234 R 17030235 R 17030236 R 17030237 R 17030228 R 17030229 R 17030230 R 17030218 R 17030219 R 17030220 R 17030221 R 17030222 R 17030223 R 17030224 R 17030225 R 17030226 R 17030227 R parsl.providers.slurm.slurm DEBUG: Updating job 17030267 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030268 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030269 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030270 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030271 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030257 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030258 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030259 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030260 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030261 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030262 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030263 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030264 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030265 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030266 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030248 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030249 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030250 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030251 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030252 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030253 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030254 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030255 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030238 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030239 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030240 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030241 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030242 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030243 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030244 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030245 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030246 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030247 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030231 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030232 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030233 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030234 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030235 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030236 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030237 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030228 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030229 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030230 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030218 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030219 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030220 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030221 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030222 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030223 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030224 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030225 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030226 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 17030227 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 109, active_tasks = 0 parsl.dataflow.strategy DEBUG: Executor multi has 0 active tasks, 109/0 running/pending blocks, and 481 connected workers parsl.dataflow.strategy DEBUG: Strategy case 1: Executor has no active tasks parsl.dataflow.strategy DEBUG: Strategy case 1b: Executor has no active tasks, and more (109) than minimum blocks (0) parsl.dataflow.strategy DEBUG: Starting idle timer for executor. If idle time exceeds 120.0s, blocks will be scaled in parsl.dataflow.strategy DEBUG: Idle time 2.86102294921875e-06s is less than max_idletime 120.0s for executor multi; not scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-22569262327184 parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=109, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 109 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91bc52391488 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91bc52391488 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cf8387c73550 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cf8387c73550 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d93f5115bff3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d93f5115bff3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8037f0fba72d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8037f0fba72d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6cbfda985a60 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6cbfda985a60 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce1734854695 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce1734854695 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf0deba7e45d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf0deba7e45d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ff8e14189298 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ff8e14189298 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fe54c630ecfb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fe54c630ecfb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c1cc8adb52d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c1cc8adb52d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 52edae18ddf3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 52edae18ddf3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fee29e324ed0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fee29e324ed0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 73856fc7d197 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 73856fc7d197 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7f75b3d79c3a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7f75b3d79c3a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: defd1da33942 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: defd1da33942 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 274845b5f660 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 274845b5f660 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2d23f1c9835a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2d23f1c9835a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c8df04ba5eb5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c8df04ba5eb5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1859ffbb00d6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1859ffbb00d6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 127189acd26d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 127189acd26d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5b7a7403c272 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5b7a7403c272 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b1ea5eb7d20c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b1ea5eb7d20c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4e88fe2a5c7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4e88fe2a5c7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a0899f3cf830 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a0899f3cf830 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e356a98bb62f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e356a98bb62f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b19809f02bf7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b19809f02bf7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6205a7ca3356 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6205a7ca3356 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84182e3826cc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84182e3826cc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a029a9a669b3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a029a9a669b3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a45d3b29a64b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a45d3b29a64b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0a684e3965e8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0a684e3965e8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: dfc023cc0267 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: dfc023cc0267 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ee5e04adfae0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ee5e04adfae0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbd92e80d20f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbd92e80d20f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ef313282b42 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ef313282b42 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9814153068c4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9814153068c4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2ebd783b687 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2ebd783b687 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f368e0941ce0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f368e0941ce0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 904579200005 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 904579200005 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3d27464fe0ba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3d27464fe0ba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5812dd12df30 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5812dd12df30 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a039e055eb8a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a039e055eb8a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b4f517823ece parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b4f517823ece parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 500633b409ba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 500633b409ba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 19b941eb7668 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 19b941eb7668 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7b99947bbb6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7b99947bbb6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0277defab17c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0277defab17c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 433129c0221d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 433129c0221d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c55a39c376f4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c55a39c376f4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a26d203713df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a26d203713df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6106229ff84e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6106229ff84e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 057e678b7209 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 057e678b7209 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fa90592ddf0a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fa90592ddf0a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 39eb37eb589b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 39eb37eb589b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 67eea277dbf2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 67eea277dbf2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4d50e445806b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4d50e445806b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3ea51abe194d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3ea51abe194d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6489b9cadd69 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6489b9cadd69 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 583f96e6a632 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 583f96e6a632 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2dc3f3b51de6 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2dc3f3b51de6 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 83f1a30f6af1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 83f1a30f6af1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 98bb230fbf6f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 98bb230fbf6f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6842bb8674c5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6842bb8674c5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2afee1566e2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2afee1566e2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5a4b669f42b5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5a4b669f42b5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ad9f2ed990c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ad9f2ed990c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 88ae2efb2013 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 88ae2efb2013 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2e8080f717bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2e8080f717bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a94630e9209f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a94630e9209f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cb2aa21fe1ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cb2aa21fe1ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce22b7136fb9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce22b7136fb9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b2c1f07268fe parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b2c1f07268fe parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d4635a5ff921 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d4635a5ff921 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ce57c6d50b94 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ce57c6d50b94 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c6d31db18d62 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c6d31db18d62 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba824131b3bf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba824131b3bf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 11690acf45e0 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 11690acf45e0 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 58d9ded7d79f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 58d9ded7d79f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: a7f3337cf91a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: a7f3337cf91a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c3de2b18432e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c3de2b18432e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8f7d65090829 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8f7d65090829 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 43fd0c29024a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 43fd0c29024a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ea206c34fe1c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ea206c34fe1c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 61dd3663adb8 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 61dd3663adb8 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 56915461471d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 56915461471d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e16ee33d70d1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e16ee33d70d1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e9f68aeb365a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e9f68aeb365a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 74e091a700bc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 74e091a700bc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1813fca28962 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1813fca28962 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: adb997259320 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: adb997259320 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b97b0561b11f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b97b0561b11f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9abfb655a0a2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9abfb655a0a2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f7358dbdbbec parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f7358dbdbbec parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 7482e191ae34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 7482e191ae34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b14f6d55d2cb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b14f6d55d2cb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e918822047c3 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e918822047c3 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fdd7442e840 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fdd7442e840 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9132c522199d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9132c522199d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6fe2e5b4f32c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6fe2e5b4f32c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 10af38f34223 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 10af38f34223 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6be0d3ee6f1b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6be0d3ee6f1b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d18b2105965d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d18b2105965d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f5cd0ca15094 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f5cd0ca15094 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: cdaf4c6e6b72 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: cdaf4c6e6b72 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4f28eca37293 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4f28eca37293 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 30ba0f1e3b6b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 30ba0f1e3b6b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d566d8ace689 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d566d8ace689 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 84803429ae1b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 84803429ae1b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5058983d4773 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5058983d4773 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030242 in provider checkpoint proc_lsst.multi INFO: found job 17030242 in provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:201 [INFO] cancelling 17030242 on provider checkpoint proc_lsst.multi INFO: cancelling 17030242 on provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030246 in provider checkpoint proc_lsst.multi INFO: found job 17030246 in provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:201 [INFO] cancelling 17030246 on provider checkpoint proc_lsst.multi INFO: cancelling 17030246 on provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030247 in provider checkpoint proc_lsst.multi INFO: found job 17030247 in provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:201 [INFO] cancelling 17030247 on provider checkpoint proc_lsst.multi INFO: cancelling 17030247 on provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:146 [INFO] found job 17030248 in provider checkpoint proc_lsst.multi INFO: found job 17030248 in provider checkpoint 2024-03-14 04:34:32 proc_lsst.multi:201 [INFO] cancelling 17030248 on provider checkpoint proc_lsst.multi INFO: cancelling 17030248 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030241 in provider checkpoint proc_lsst.multi INFO: found job 17030241 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030241 on provider checkpoint proc_lsst.multi INFO: cancelling 17030241 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030244 in provider checkpoint proc_lsst.multi INFO: found job 17030244 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030244 on provider checkpoint proc_lsst.multi INFO: cancelling 17030244 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030243 in provider checkpoint proc_lsst.multi INFO: found job 17030243 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030243 on provider checkpoint proc_lsst.multi INFO: cancelling 17030243 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030261 in provider checkpoint proc_lsst.multi INFO: found job 17030261 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030261 on provider checkpoint proc_lsst.multi INFO: cancelling 17030261 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030263 in provider checkpoint proc_lsst.multi INFO: found job 17030263 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030263 on provider checkpoint proc_lsst.multi INFO: cancelling 17030263 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030252 in provider checkpoint proc_lsst.multi INFO: found job 17030252 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030252 on provider checkpoint proc_lsst.multi INFO: cancelling 17030252 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030233 in provider checkpoint proc_lsst.multi INFO: found job 17030233 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030233 on provider checkpoint proc_lsst.multi INFO: cancelling 17030233 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030232 in provider checkpoint proc_lsst.multi INFO: found job 17030232 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030232 on provider checkpoint proc_lsst.multi INFO: cancelling 17030232 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030234 in provider checkpoint proc_lsst.multi INFO: found job 17030234 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030234 on provider checkpoint proc_lsst.multi INFO: cancelling 17030234 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030251 in provider checkpoint proc_lsst.multi INFO: found job 17030251 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030251 on provider checkpoint proc_lsst.multi INFO: cancelling 17030251 on provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:146 [INFO] found job 17030253 in provider checkpoint proc_lsst.multi INFO: found job 17030253 in provider checkpoint 2024-03-14 04:34:33 proc_lsst.multi:201 [INFO] cancelling 17030253 on provider checkpoint proc_lsst.multi INFO: cancelling 17030253 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030260 in provider checkpoint proc_lsst.multi INFO: found job 17030260 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030260 on provider checkpoint proc_lsst.multi INFO: cancelling 17030260 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030250 in provider checkpoint proc_lsst.multi INFO: found job 17030250 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030250 on provider checkpoint proc_lsst.multi INFO: cancelling 17030250 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030259 in provider checkpoint proc_lsst.multi INFO: found job 17030259 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030259 on provider checkpoint proc_lsst.multi INFO: cancelling 17030259 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030249 in provider checkpoint proc_lsst.multi INFO: found job 17030249 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030249 on provider checkpoint proc_lsst.multi INFO: cancelling 17030249 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030266 in provider checkpoint proc_lsst.multi INFO: found job 17030266 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030266 on provider checkpoint proc_lsst.multi INFO: cancelling 17030266 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030257 in provider checkpoint proc_lsst.multi INFO: found job 17030257 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030257 on provider checkpoint proc_lsst.multi INFO: cancelling 17030257 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030258 in provider checkpoint proc_lsst.multi INFO: found job 17030258 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030258 on provider checkpoint proc_lsst.multi INFO: cancelling 17030258 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030245 in provider checkpoint proc_lsst.multi INFO: found job 17030245 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030245 on provider checkpoint proc_lsst.multi INFO: cancelling 17030245 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030267 in provider checkpoint proc_lsst.multi INFO: found job 17030267 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030267 on provider checkpoint proc_lsst.multi INFO: cancelling 17030267 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030265 in provider checkpoint proc_lsst.multi INFO: found job 17030265 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030265 on provider checkpoint proc_lsst.multi INFO: cancelling 17030265 on provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:146 [INFO] found job 17030264 in provider checkpoint proc_lsst.multi INFO: found job 17030264 in provider checkpoint 2024-03-14 04:34:34 proc_lsst.multi:201 [INFO] cancelling 17030264 on provider checkpoint proc_lsst.multi INFO: cancelling 17030264 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030231 in provider checkpoint proc_lsst.multi INFO: found job 17030231 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030231 on provider checkpoint proc_lsst.multi INFO: cancelling 17030231 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030255 in provider checkpoint proc_lsst.multi INFO: found job 17030255 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030255 on provider checkpoint proc_lsst.multi INFO: cancelling 17030255 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030254 in provider checkpoint proc_lsst.multi INFO: found job 17030254 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030254 on provider checkpoint proc_lsst.multi INFO: cancelling 17030254 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030271 in provider checkpoint proc_lsst.multi INFO: found job 17030271 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030271 on provider checkpoint proc_lsst.multi INFO: cancelling 17030271 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030223 in provider checkpoint proc_lsst.multi INFO: found job 17030223 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030223 on provider checkpoint proc_lsst.multi INFO: cancelling 17030223 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030238 in provider checkpoint proc_lsst.multi INFO: found job 17030238 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030238 on provider checkpoint proc_lsst.multi INFO: cancelling 17030238 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030239 in provider checkpoint proc_lsst.multi INFO: found job 17030239 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030239 on provider checkpoint proc_lsst.multi INFO: cancelling 17030239 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030235 in provider checkpoint proc_lsst.multi INFO: found job 17030235 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030235 on provider checkpoint proc_lsst.multi INFO: cancelling 17030235 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030236 in provider checkpoint proc_lsst.multi INFO: found job 17030236 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030236 on provider checkpoint proc_lsst.multi INFO: cancelling 17030236 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030237 in provider checkpoint proc_lsst.multi INFO: found job 17030237 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030237 on provider checkpoint proc_lsst.multi INFO: cancelling 17030237 on provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:146 [INFO] found job 17030226 in provider checkpoint proc_lsst.multi INFO: found job 17030226 in provider checkpoint 2024-03-14 04:34:35 proc_lsst.multi:201 [INFO] cancelling 17030226 on provider checkpoint proc_lsst.multi INFO: cancelling 17030226 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030219 in provider checkpoint proc_lsst.multi INFO: found job 17030219 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030219 on provider checkpoint proc_lsst.multi INFO: cancelling 17030219 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030269 in provider checkpoint proc_lsst.multi INFO: found job 17030269 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030269 on provider checkpoint proc_lsst.multi INFO: cancelling 17030269 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030224 in provider checkpoint proc_lsst.multi INFO: found job 17030224 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030224 on provider checkpoint proc_lsst.multi INFO: cancelling 17030224 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030230 in provider checkpoint proc_lsst.multi INFO: found job 17030230 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030230 on provider checkpoint proc_lsst.multi INFO: cancelling 17030230 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030268 in provider checkpoint proc_lsst.multi INFO: found job 17030268 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030268 on provider checkpoint proc_lsst.multi INFO: cancelling 17030268 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030228 in provider checkpoint proc_lsst.multi INFO: found job 17030228 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030228 on provider checkpoint proc_lsst.multi INFO: cancelling 17030228 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030225 in provider checkpoint proc_lsst.multi INFO: found job 17030225 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030225 on provider checkpoint proc_lsst.multi INFO: cancelling 17030225 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030221 in provider checkpoint proc_lsst.multi INFO: found job 17030221 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030221 on provider checkpoint proc_lsst.multi INFO: cancelling 17030221 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030240 in provider checkpoint proc_lsst.multi INFO: found job 17030240 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030240 on provider checkpoint proc_lsst.multi INFO: cancelling 17030240 on provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:146 [INFO] found job 17030218 in provider checkpoint proc_lsst.multi INFO: found job 17030218 in provider checkpoint 2024-03-14 04:34:36 proc_lsst.multi:201 [INFO] cancelling 17030218 on provider checkpoint proc_lsst.multi INFO: cancelling 17030218 on provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030229 in provider checkpoint proc_lsst.multi INFO: found job 17030229 in provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030229 on provider checkpoint proc_lsst.multi INFO: cancelling 17030229 on provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030220 in provider checkpoint proc_lsst.multi INFO: found job 17030220 in provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030220 on provider checkpoint proc_lsst.multi INFO: cancelling 17030220 on provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030222 in provider checkpoint proc_lsst.multi INFO: found job 17030222 in provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030222 on provider checkpoint proc_lsst.multi INFO: cancelling 17030222 on provider checkpoint 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030192 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030192 in provider escience_rtx6k 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030192 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030192 on provider escience_rtx6k 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030165 in provider astro proc_lsst.multi INFO: found job 17030165 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030165 on provider astro proc_lsst.multi INFO: cancelling 17030165 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030169 in provider astro proc_lsst.multi INFO: found job 17030169 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030169 on provider astro proc_lsst.multi INFO: cancelling 17030169 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030164 in provider astro proc_lsst.multi INFO: found job 17030164 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030164 on provider astro proc_lsst.multi INFO: cancelling 17030164 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030174 in provider astro proc_lsst.multi INFO: found job 17030174 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030174 on provider astro proc_lsst.multi INFO: cancelling 17030174 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030180 in provider astro proc_lsst.multi INFO: found job 17030180 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030180 on provider astro proc_lsst.multi INFO: cancelling 17030180 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030178 in provider astro proc_lsst.multi INFO: found job 17030178 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030178 on provider astro proc_lsst.multi INFO: cancelling 17030178 on provider astro 2024-03-14 04:34:37 proc_lsst.multi:146 [INFO] found job 17030167 in provider astro proc_lsst.multi INFO: found job 17030167 in provider astro 2024-03-14 04:34:37 proc_lsst.multi:201 [INFO] cancelling 17030167 on provider astro proc_lsst.multi INFO: cancelling 17030167 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030176 in provider astro proc_lsst.multi INFO: found job 17030176 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030176 on provider astro proc_lsst.multi INFO: cancelling 17030176 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030172 in provider astro proc_lsst.multi INFO: found job 17030172 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030172 on provider astro proc_lsst.multi INFO: cancelling 17030172 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030177 in provider astro proc_lsst.multi INFO: found job 17030177 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030177 on provider astro proc_lsst.multi INFO: cancelling 17030177 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030173 in provider astro proc_lsst.multi INFO: found job 17030173 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030173 on provider astro proc_lsst.multi INFO: cancelling 17030173 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030171 in provider astro proc_lsst.multi INFO: found job 17030171 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030171 on provider astro proc_lsst.multi INFO: cancelling 17030171 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030181 in provider astro proc_lsst.multi INFO: found job 17030181 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030181 on provider astro proc_lsst.multi INFO: cancelling 17030181 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030170 in provider astro proc_lsst.multi INFO: found job 17030170 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030170 on provider astro proc_lsst.multi INFO: cancelling 17030170 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030163 in provider astro proc_lsst.multi INFO: found job 17030163 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030163 on provider astro proc_lsst.multi INFO: cancelling 17030163 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030166 in provider astro proc_lsst.multi INFO: found job 17030166 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030166 on provider astro proc_lsst.multi INFO: cancelling 17030166 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030168 in provider astro proc_lsst.multi INFO: found job 17030168 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030168 on provider astro proc_lsst.multi INFO: cancelling 17030168 on provider astro 2024-03-14 04:34:38 proc_lsst.multi:146 [INFO] found job 17030179 in provider astro proc_lsst.multi INFO: found job 17030179 in provider astro 2024-03-14 04:34:38 proc_lsst.multi:201 [INFO] cancelling 17030179 on provider astro proc_lsst.multi INFO: cancelling 17030179 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030175 in provider astro proc_lsst.multi INFO: found job 17030175 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030175 on provider astro proc_lsst.multi INFO: cancelling 17030175 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030162 in provider astro proc_lsst.multi INFO: found job 17030162 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030162 on provider astro proc_lsst.multi INFO: cancelling 17030162 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030209 in provider escience_a40 proc_lsst.multi INFO: found job 17030209 in provider escience_a40 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030209 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030209 on provider escience_a40 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030184 in provider astro proc_lsst.multi INFO: found job 17030184 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030184 on provider astro proc_lsst.multi INFO: cancelling 17030184 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030188 in provider astro proc_lsst.multi INFO: found job 17030188 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030188 on provider astro proc_lsst.multi INFO: cancelling 17030188 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030186 in provider astro proc_lsst.multi INFO: found job 17030186 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030186 on provider astro proc_lsst.multi INFO: cancelling 17030186 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030187 in provider astro proc_lsst.multi INFO: found job 17030187 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030187 on provider astro proc_lsst.multi INFO: cancelling 17030187 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030185 in provider astro proc_lsst.multi INFO: found job 17030185 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030185 on provider astro proc_lsst.multi INFO: cancelling 17030185 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030182 in provider astro proc_lsst.multi INFO: found job 17030182 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030182 on provider astro proc_lsst.multi INFO: cancelling 17030182 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030191 in provider astro proc_lsst.multi INFO: found job 17030191 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030191 on provider astro proc_lsst.multi INFO: cancelling 17030191 on provider astro 2024-03-14 04:34:39 proc_lsst.multi:146 [INFO] found job 17030189 in provider astro proc_lsst.multi INFO: found job 17030189 in provider astro 2024-03-14 04:34:39 proc_lsst.multi:201 [INFO] cancelling 17030189 on provider astro proc_lsst.multi INFO: cancelling 17030189 on provider astro 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030183 in provider astro proc_lsst.multi INFO: found job 17030183 in provider astro 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030183 on provider astro proc_lsst.multi INFO: cancelling 17030183 on provider astro 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030190 in provider astro proc_lsst.multi INFO: found job 17030190 in provider astro 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030190 on provider astro proc_lsst.multi INFO: cancelling 17030190 on provider astro 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030203 in provider escience_a40 proc_lsst.multi INFO: found job 17030203 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030203 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030203 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030200 in provider escience_a40 proc_lsst.multi INFO: found job 17030200 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030200 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030200 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030205 in provider escience_a40 proc_lsst.multi INFO: found job 17030205 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030205 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030205 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030197 in provider escience_a40 proc_lsst.multi INFO: found job 17030197 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030197 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030197 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030201 in provider escience_a40 proc_lsst.multi INFO: found job 17030201 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030201 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030201 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030207 in provider escience_a40 proc_lsst.multi INFO: found job 17030207 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030207 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030207 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030199 in provider escience_a40 proc_lsst.multi INFO: found job 17030199 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030199 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030199 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030204 in provider escience_a40 proc_lsst.multi INFO: found job 17030204 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030204 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030204 on provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:146 [INFO] found job 17030198 in provider escience_a40 proc_lsst.multi INFO: found job 17030198 in provider escience_a40 2024-03-14 04:34:40 proc_lsst.multi:201 [INFO] cancelling 17030198 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030198 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030202 in provider escience_a40 proc_lsst.multi INFO: found job 17030202 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030202 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030202 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030206 in provider escience_a40 proc_lsst.multi INFO: found job 17030206 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030206 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030206 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030210 in provider escience_a40 proc_lsst.multi INFO: found job 17030210 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030210 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030213 in provider escience_a40 proc_lsst.multi INFO: found job 17030213 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030213 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030214 in provider escience_a40 proc_lsst.multi INFO: found job 17030214 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030214 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030211 in provider escience_a40 proc_lsst.multi INFO: found job 17030211 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030211 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030208 in provider escience_a40 proc_lsst.multi INFO: found job 17030208 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030208 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030208 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030217 in provider escience_a40 proc_lsst.multi INFO: found job 17030217 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030217 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030215 in provider escience_a40 proc_lsst.multi INFO: found job 17030215 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030215 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030216 in provider escience_a40 proc_lsst.multi INFO: found job 17030216 in provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030216 on provider escience_a40 2024-03-14 04:34:41 proc_lsst.multi:146 [INFO] found job 17030227 in provider checkpoint proc_lsst.multi INFO: found job 17030227 in provider checkpoint 2024-03-14 04:34:41 proc_lsst.multi:201 [INFO] cancelling 17030227 on provider checkpoint proc_lsst.multi INFO: cancelling 17030227 on provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 98211 in provider local proc_lsst.multi INFO: found job 98211 in provider local 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 98211 on provider local proc_lsst.multi INFO: cancelling 98211 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 98211 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030195 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030195 in provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030195 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030195 on provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030193 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030193 in provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030193 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030193 on provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030262 in provider checkpoint proc_lsst.multi INFO: found job 17030262 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030262 on provider checkpoint proc_lsst.multi INFO: cancelling 17030262 on provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030194 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030194 in provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030194 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030194 on provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030196 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030196 in provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030196 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030196 on provider escience_rtx6k 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030270 in provider checkpoint proc_lsst.multi INFO: found job 17030270 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030270 on provider checkpoint proc_lsst.multi INFO: cancelling 17030270 on provider checkpoint parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-14 04:34:42 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-14 04:34:42 proc_lsst.multi:47 [INFO] new jobs since last cancel ['17030218', '17030219', '17030220', '17030221', '17030222', '17030223', '17030224', '17030225', '17030226', '17030227', '17030228', '17030229', '17030230', '17030231', '17030232', '17030233', '17030234', '17030235', '17030236', '17030237', '17030238', '17030239', '17030240', '17030241', '17030242', '17030243', '17030244', '17030245', '17030246', '17030247', '17030248', '17030249', '17030250', '17030251', '17030252', '17030253', '17030254', '17030255', '17030257', '17030258', '17030259', '17030260', '17030261', '17030262', '17030263', '17030264', '17030265', '17030266', '17030267', '17030268', '17030269', '17030270', '17030271', '17030162', '17030163', '17030164', '17030165', '17030166', '17030167', '17030168', '17030169', '17030170', '17030171', '17030172', '17030173', '17030174', '17030175', '17030176', '17030177', '17030178', '17030179', '17030180', '17030181', '17030182', '17030183', '17030184', '17030185', '17030186', '17030187', '17030188', '17030189', '17030190', '17030191', '17030197', '17030198', '17030199', '17030200', '17030201', '17030202', '17030203', '17030204', '17030205', '17030206', '17030207', '17030208', '17030209', '17030210', '17030211', '17030213', '17030214', '17030215', '17030216', '17030217', '17030192', '17030193', '17030194', '17030195', '17030196', '98211'] proc_lsst.multi INFO: new jobs since last cancel ['17030218', '17030219', '17030220', '17030221', '17030222', '17030223', '17030224', '17030225', '17030226', '17030227', '17030228', '17030229', '17030230', '17030231', '17030232', '17030233', '17030234', '17030235', '17030236', '17030237', '17030238', '17030239', '17030240', '17030241', '17030242', '17030243', '17030244', '17030245', '17030246', '17030247', '17030248', '17030249', '17030250', '17030251', '17030252', '17030253', '17030254', '17030255', '17030257', '17030258', '17030259', '17030260', '17030261', '17030262', '17030263', '17030264', '17030265', '17030266', '17030267', '17030268', '17030269', '17030270', '17030271', '17030162', '17030163', '17030164', '17030165', '17030166', '17030167', '17030168', '17030169', '17030170', '17030171', '17030172', '17030173', '17030174', '17030175', '17030176', '17030177', '17030178', '17030179', '17030180', '17030181', '17030182', '17030183', '17030184', '17030185', '17030186', '17030187', '17030188', '17030189', '17030190', '17030191', '17030197', '17030198', '17030199', '17030200', '17030201', '17030202', '17030203', '17030204', '17030205', '17030206', '17030207', '17030208', '17030209', '17030210', '17030211', '17030213', '17030214', '17030215', '17030216', '17030217', '17030192', '17030193', '17030194', '17030195', '17030196', '98211'] 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030218 in provider checkpoint proc_lsst.multi INFO: found job 17030218 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030218 on provider checkpoint proc_lsst.multi INFO: cancelling 17030218 on provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030219 in provider checkpoint proc_lsst.multi INFO: found job 17030219 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030219 on provider checkpoint proc_lsst.multi INFO: cancelling 17030219 on provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030220 in provider checkpoint proc_lsst.multi INFO: found job 17030220 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030220 on provider checkpoint proc_lsst.multi INFO: cancelling 17030220 on provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:146 [INFO] found job 17030221 in provider checkpoint proc_lsst.multi INFO: found job 17030221 in provider checkpoint 2024-03-14 04:34:42 proc_lsst.multi:201 [INFO] cancelling 17030221 on provider checkpoint proc_lsst.multi INFO: cancelling 17030221 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030222 in provider checkpoint proc_lsst.multi INFO: found job 17030222 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030222 on provider checkpoint proc_lsst.multi INFO: cancelling 17030222 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030223 in provider checkpoint proc_lsst.multi INFO: found job 17030223 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030223 on provider checkpoint proc_lsst.multi INFO: cancelling 17030223 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030224 in provider checkpoint proc_lsst.multi INFO: found job 17030224 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030224 on provider checkpoint proc_lsst.multi INFO: cancelling 17030224 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030225 in provider checkpoint proc_lsst.multi INFO: found job 17030225 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030225 on provider checkpoint proc_lsst.multi INFO: cancelling 17030225 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030226 in provider checkpoint proc_lsst.multi INFO: found job 17030226 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030226 on provider checkpoint proc_lsst.multi INFO: cancelling 17030226 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030227 in provider checkpoint proc_lsst.multi INFO: found job 17030227 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030227 on provider checkpoint proc_lsst.multi INFO: cancelling 17030227 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030228 in provider checkpoint proc_lsst.multi INFO: found job 17030228 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030228 on provider checkpoint proc_lsst.multi INFO: cancelling 17030228 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030229 in provider checkpoint proc_lsst.multi INFO: found job 17030229 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030229 on provider checkpoint proc_lsst.multi INFO: cancelling 17030229 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030230 in provider checkpoint proc_lsst.multi INFO: found job 17030230 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030230 on provider checkpoint proc_lsst.multi INFO: cancelling 17030230 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030231 in provider checkpoint proc_lsst.multi INFO: found job 17030231 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030231 on provider checkpoint proc_lsst.multi INFO: cancelling 17030231 on provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:146 [INFO] found job 17030232 in provider checkpoint proc_lsst.multi INFO: found job 17030232 in provider checkpoint 2024-03-14 04:34:43 proc_lsst.multi:201 [INFO] cancelling 17030232 on provider checkpoint proc_lsst.multi INFO: cancelling 17030232 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030233 in provider checkpoint proc_lsst.multi INFO: found job 17030233 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030233 on provider checkpoint proc_lsst.multi INFO: cancelling 17030233 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030234 in provider checkpoint proc_lsst.multi INFO: found job 17030234 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030234 on provider checkpoint proc_lsst.multi INFO: cancelling 17030234 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030235 in provider checkpoint proc_lsst.multi INFO: found job 17030235 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030235 on provider checkpoint proc_lsst.multi INFO: cancelling 17030235 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030236 in provider checkpoint proc_lsst.multi INFO: found job 17030236 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030236 on provider checkpoint proc_lsst.multi INFO: cancelling 17030236 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030237 in provider checkpoint proc_lsst.multi INFO: found job 17030237 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030237 on provider checkpoint proc_lsst.multi INFO: cancelling 17030237 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030238 in provider checkpoint proc_lsst.multi INFO: found job 17030238 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030238 on provider checkpoint proc_lsst.multi INFO: cancelling 17030238 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030239 in provider checkpoint proc_lsst.multi INFO: found job 17030239 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030239 on provider checkpoint proc_lsst.multi INFO: cancelling 17030239 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030240 in provider checkpoint proc_lsst.multi INFO: found job 17030240 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030240 on provider checkpoint proc_lsst.multi INFO: cancelling 17030240 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030241 in provider checkpoint proc_lsst.multi INFO: found job 17030241 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030241 on provider checkpoint proc_lsst.multi INFO: cancelling 17030241 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030242 in provider checkpoint proc_lsst.multi INFO: found job 17030242 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030242 on provider checkpoint proc_lsst.multi INFO: cancelling 17030242 on provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:146 [INFO] found job 17030243 in provider checkpoint proc_lsst.multi INFO: found job 17030243 in provider checkpoint 2024-03-14 04:34:44 proc_lsst.multi:201 [INFO] cancelling 17030243 on provider checkpoint proc_lsst.multi INFO: cancelling 17030243 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030244 in provider checkpoint proc_lsst.multi INFO: found job 17030244 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030244 on provider checkpoint proc_lsst.multi INFO: cancelling 17030244 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030245 in provider checkpoint proc_lsst.multi INFO: found job 17030245 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030245 on provider checkpoint proc_lsst.multi INFO: cancelling 17030245 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030246 in provider checkpoint proc_lsst.multi INFO: found job 17030246 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030246 on provider checkpoint proc_lsst.multi INFO: cancelling 17030246 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030247 in provider checkpoint proc_lsst.multi INFO: found job 17030247 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030247 on provider checkpoint proc_lsst.multi INFO: cancelling 17030247 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030248 in provider checkpoint proc_lsst.multi INFO: found job 17030248 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030248 on provider checkpoint proc_lsst.multi INFO: cancelling 17030248 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030249 in provider checkpoint proc_lsst.multi INFO: found job 17030249 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030249 on provider checkpoint proc_lsst.multi INFO: cancelling 17030249 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030250 in provider checkpoint proc_lsst.multi INFO: found job 17030250 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030250 on provider checkpoint proc_lsst.multi INFO: cancelling 17030250 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030251 in provider checkpoint proc_lsst.multi INFO: found job 17030251 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030251 on provider checkpoint proc_lsst.multi INFO: cancelling 17030251 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030252 in provider checkpoint proc_lsst.multi INFO: found job 17030252 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030252 on provider checkpoint proc_lsst.multi INFO: cancelling 17030252 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030253 in provider checkpoint proc_lsst.multi INFO: found job 17030253 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030253 on provider checkpoint proc_lsst.multi INFO: cancelling 17030253 on provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:146 [INFO] found job 17030254 in provider checkpoint proc_lsst.multi INFO: found job 17030254 in provider checkpoint 2024-03-14 04:34:45 proc_lsst.multi:201 [INFO] cancelling 17030254 on provider checkpoint proc_lsst.multi INFO: cancelling 17030254 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030255 in provider checkpoint proc_lsst.multi INFO: found job 17030255 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030255 on provider checkpoint proc_lsst.multi INFO: cancelling 17030255 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030257 in provider checkpoint proc_lsst.multi INFO: found job 17030257 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030257 on provider checkpoint proc_lsst.multi INFO: cancelling 17030257 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030258 in provider checkpoint proc_lsst.multi INFO: found job 17030258 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030258 on provider checkpoint proc_lsst.multi INFO: cancelling 17030258 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030259 in provider checkpoint proc_lsst.multi INFO: found job 17030259 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030259 on provider checkpoint proc_lsst.multi INFO: cancelling 17030259 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030260 in provider checkpoint proc_lsst.multi INFO: found job 17030260 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030260 on provider checkpoint proc_lsst.multi INFO: cancelling 17030260 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030261 in provider checkpoint proc_lsst.multi INFO: found job 17030261 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030261 on provider checkpoint proc_lsst.multi INFO: cancelling 17030261 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030262 in provider checkpoint proc_lsst.multi INFO: found job 17030262 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030262 on provider checkpoint proc_lsst.multi INFO: cancelling 17030262 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030263 in provider checkpoint proc_lsst.multi INFO: found job 17030263 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030263 on provider checkpoint proc_lsst.multi INFO: cancelling 17030263 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030264 in provider checkpoint proc_lsst.multi INFO: found job 17030264 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030264 on provider checkpoint proc_lsst.multi INFO: cancelling 17030264 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030265 in provider checkpoint proc_lsst.multi INFO: found job 17030265 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030265 on provider checkpoint proc_lsst.multi INFO: cancelling 17030265 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030266 in provider checkpoint proc_lsst.multi INFO: found job 17030266 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030266 on provider checkpoint proc_lsst.multi INFO: cancelling 17030266 on provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:146 [INFO] found job 17030267 in provider checkpoint proc_lsst.multi INFO: found job 17030267 in provider checkpoint 2024-03-14 04:34:46 proc_lsst.multi:201 [INFO] cancelling 17030267 on provider checkpoint proc_lsst.multi INFO: cancelling 17030267 on provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030268 in provider checkpoint proc_lsst.multi INFO: found job 17030268 in provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030268 on provider checkpoint proc_lsst.multi INFO: cancelling 17030268 on provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030269 in provider checkpoint proc_lsst.multi INFO: found job 17030269 in provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030269 on provider checkpoint proc_lsst.multi INFO: cancelling 17030269 on provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030270 in provider checkpoint proc_lsst.multi INFO: found job 17030270 in provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030270 on provider checkpoint proc_lsst.multi INFO: cancelling 17030270 on provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030271 in provider checkpoint proc_lsst.multi INFO: found job 17030271 in provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030271 on provider checkpoint proc_lsst.multi INFO: cancelling 17030271 on provider checkpoint 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030162 in provider astro proc_lsst.multi INFO: found job 17030162 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030162 on provider astro proc_lsst.multi INFO: cancelling 17030162 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030163 in provider astro proc_lsst.multi INFO: found job 17030163 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030163 on provider astro proc_lsst.multi INFO: cancelling 17030163 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030164 in provider astro proc_lsst.multi INFO: found job 17030164 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030164 on provider astro proc_lsst.multi INFO: cancelling 17030164 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030165 in provider astro proc_lsst.multi INFO: found job 17030165 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030165 on provider astro proc_lsst.multi INFO: cancelling 17030165 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030166 in provider astro proc_lsst.multi INFO: found job 17030166 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030166 on provider astro proc_lsst.multi INFO: cancelling 17030166 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030167 in provider astro proc_lsst.multi INFO: found job 17030167 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030167 on provider astro proc_lsst.multi INFO: cancelling 17030167 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030168 in provider astro proc_lsst.multi INFO: found job 17030168 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030168 on provider astro proc_lsst.multi INFO: cancelling 17030168 on provider astro 2024-03-14 04:34:47 proc_lsst.multi:146 [INFO] found job 17030169 in provider astro proc_lsst.multi INFO: found job 17030169 in provider astro 2024-03-14 04:34:47 proc_lsst.multi:201 [INFO] cancelling 17030169 on provider astro proc_lsst.multi INFO: cancelling 17030169 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030170 in provider astro proc_lsst.multi INFO: found job 17030170 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030170 on provider astro proc_lsst.multi INFO: cancelling 17030170 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030171 in provider astro proc_lsst.multi INFO: found job 17030171 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030171 on provider astro proc_lsst.multi INFO: cancelling 17030171 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030172 in provider astro proc_lsst.multi INFO: found job 17030172 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030172 on provider astro proc_lsst.multi INFO: cancelling 17030172 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030173 in provider astro proc_lsst.multi INFO: found job 17030173 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030173 on provider astro proc_lsst.multi INFO: cancelling 17030173 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030174 in provider astro proc_lsst.multi INFO: found job 17030174 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030174 on provider astro proc_lsst.multi INFO: cancelling 17030174 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030175 in provider astro proc_lsst.multi INFO: found job 17030175 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030175 on provider astro proc_lsst.multi INFO: cancelling 17030175 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030176 in provider astro proc_lsst.multi INFO: found job 17030176 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030176 on provider astro proc_lsst.multi INFO: cancelling 17030176 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030177 in provider astro proc_lsst.multi INFO: found job 17030177 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030177 on provider astro proc_lsst.multi INFO: cancelling 17030177 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030178 in provider astro proc_lsst.multi INFO: found job 17030178 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030178 on provider astro proc_lsst.multi INFO: cancelling 17030178 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030179 in provider astro proc_lsst.multi INFO: found job 17030179 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030179 on provider astro proc_lsst.multi INFO: cancelling 17030179 on provider astro 2024-03-14 04:34:48 proc_lsst.multi:146 [INFO] found job 17030180 in provider astro proc_lsst.multi INFO: found job 17030180 in provider astro 2024-03-14 04:34:48 proc_lsst.multi:201 [INFO] cancelling 17030180 on provider astro proc_lsst.multi INFO: cancelling 17030180 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030181 in provider astro proc_lsst.multi INFO: found job 17030181 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030181 on provider astro proc_lsst.multi INFO: cancelling 17030181 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030182 in provider astro proc_lsst.multi INFO: found job 17030182 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030182 on provider astro proc_lsst.multi INFO: cancelling 17030182 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030183 in provider astro proc_lsst.multi INFO: found job 17030183 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030183 on provider astro proc_lsst.multi INFO: cancelling 17030183 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030184 in provider astro proc_lsst.multi INFO: found job 17030184 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030184 on provider astro proc_lsst.multi INFO: cancelling 17030184 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030185 in provider astro proc_lsst.multi INFO: found job 17030185 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030185 on provider astro proc_lsst.multi INFO: cancelling 17030185 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030186 in provider astro proc_lsst.multi INFO: found job 17030186 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030186 on provider astro proc_lsst.multi INFO: cancelling 17030186 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030187 in provider astro proc_lsst.multi INFO: found job 17030187 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030187 on provider astro proc_lsst.multi INFO: cancelling 17030187 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030188 in provider astro proc_lsst.multi INFO: found job 17030188 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030188 on provider astro proc_lsst.multi INFO: cancelling 17030188 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030189 in provider astro proc_lsst.multi INFO: found job 17030189 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030189 on provider astro proc_lsst.multi INFO: cancelling 17030189 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030190 in provider astro proc_lsst.multi INFO: found job 17030190 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030190 on provider astro proc_lsst.multi INFO: cancelling 17030190 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030191 in provider astro proc_lsst.multi INFO: found job 17030191 in provider astro 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030191 on provider astro proc_lsst.multi INFO: cancelling 17030191 on provider astro 2024-03-14 04:34:49 proc_lsst.multi:146 [INFO] found job 17030197 in provider escience_a40 proc_lsst.multi INFO: found job 17030197 in provider escience_a40 2024-03-14 04:34:49 proc_lsst.multi:201 [INFO] cancelling 17030197 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030197 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030198 in provider escience_a40 proc_lsst.multi INFO: found job 17030198 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030198 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030198 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030199 in provider escience_a40 proc_lsst.multi INFO: found job 17030199 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030199 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030199 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030200 in provider escience_a40 proc_lsst.multi INFO: found job 17030200 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030200 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030200 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030201 in provider escience_a40 proc_lsst.multi INFO: found job 17030201 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030201 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030201 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030202 in provider escience_a40 proc_lsst.multi INFO: found job 17030202 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030202 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030202 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030203 in provider escience_a40 proc_lsst.multi INFO: found job 17030203 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030203 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030203 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030204 in provider escience_a40 proc_lsst.multi INFO: found job 17030204 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030204 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030204 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030205 in provider escience_a40 proc_lsst.multi INFO: found job 17030205 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030205 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030205 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030206 in provider escience_a40 proc_lsst.multi INFO: found job 17030206 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030206 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030206 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030207 in provider escience_a40 proc_lsst.multi INFO: found job 17030207 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030207 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030207 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030208 in provider escience_a40 proc_lsst.multi INFO: found job 17030208 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030208 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030208 on provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:146 [INFO] found job 17030209 in provider escience_a40 proc_lsst.multi INFO: found job 17030209 in provider escience_a40 2024-03-14 04:34:50 proc_lsst.multi:201 [INFO] cancelling 17030209 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030209 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030210 in provider escience_a40 proc_lsst.multi INFO: found job 17030210 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030210 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030210 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030211 in provider escience_a40 proc_lsst.multi INFO: found job 17030211 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030211 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030211 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030213 in provider escience_a40 proc_lsst.multi INFO: found job 17030213 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030213 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030213 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030214 in provider escience_a40 proc_lsst.multi INFO: found job 17030214 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030214 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030214 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030215 in provider escience_a40 proc_lsst.multi INFO: found job 17030215 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030215 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030215 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030216 in provider escience_a40 proc_lsst.multi INFO: found job 17030216 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030216 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030216 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030217 in provider escience_a40 proc_lsst.multi INFO: found job 17030217 in provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030217 on provider escience_a40 proc_lsst.multi INFO: cancelling 17030217 on provider escience_a40 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030192 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030192 in provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030192 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030192 on provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030193 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030193 in provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030193 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030193 on provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030194 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030194 in provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030194 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030194 on provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:146 [INFO] found job 17030195 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030195 in provider escience_rtx6k 2024-03-14 04:34:51 proc_lsst.multi:201 [INFO] cancelling 17030195 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030195 on provider escience_rtx6k 2024-03-14 04:34:52 proc_lsst.multi:146 [INFO] found job 17030196 in provider escience_rtx6k proc_lsst.multi INFO: found job 17030196 in provider escience_rtx6k 2024-03-14 04:34:52 proc_lsst.multi:201 [INFO] cancelling 17030196 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 17030196 on provider escience_rtx6k 2024-03-14 04:34:52 proc_lsst.multi:146 [INFO] found job 98211 in provider local proc_lsst.multi INFO: found job 98211 in provider local 2024-03-14 04:34:52 proc_lsst.multi:201 [INFO] cancelling 98211 on provider local proc_lsst.multi INFO: cancelling 98211 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 98211 parsl.providers.local.local WARNING: Failed to kill PID: 98211 and child processes on local 2024-03-14 04:34:53 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 112.8341 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20201019_B1a_science#step0_20240314T043249Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 112.8402 seconds; current memory usage: 0.281 Gibyte, delta: 0.002 Gibyte, peak delta: 0.006 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 122.8657 seconds; current memory usage: 0.281 Gibyte, delta: 0.144 Gibyte, peak delta: 0.147 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.284 Gibyte (main), 0.284 Gibyte (largest child process) Run Id: None Run Name: DEEP_20201019_B1a_science#step0_20240314T043249Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up real 2m4.687s user 0m50.146s sys 0m25.871s